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The  specialized  nature  of  technology-based  programs  creates  volumes 
of  data  of  a  magnitude  never  before  seen,  complicating  the  test  and 
evaluation  phase  of  acquisition.  This  article  provides  a  practical  solu¬ 
tion  for  reducing  network  traffic  analysis  data  while  expediting  test  and 
evaluation.  From  small  lab  testing  to  full  integration  test  events,  quality 
of  service  and  other  key  metrics  of  military  systems  and  networks  are 
evaluated.  Network  data  captured  in  standard  flow  formats  enable  scal¬ 
able  approaches  for  producing  network  traffic  analyses.  Because  of  its 
compact  representation  of  network  traffic,  flow  data  naturally  scale  well. 
Some  analyses  require  deep  packet  inspection,  but  many  can  be  calcu¬ 
lated/approximated  quickly  with  flow  data,  including  quality-of-service 
metrics  like  completion  rate  and  speed  of  service. 

Improving  Statistical  Rigor  in  Defense  Test  and  Evaluation:  Use 
of  Tolerance  Intervals  in  Designed  Experiments 

n  804  Alethea  Rucker 

Leveraging  the  use  of  statistical  methods  is  critical  in  providing  defen¬ 
sible  test  data  to  the  Department  of  Defense  Test  and  Evaluation  (T&E) 
enterprise.  This  article  investigates  statistical  tolerance  intervals  in 
designed  experiments  for  the  T&E  technical  community.  Tolerance 
intervals  are  scarcely  discussed  in  extant  literature  as  compared  to 
confidence/prediction  intervals.  The  lesser  known  tolerance  intervals 
can  ensure  a  proportion  of  the  population  is  captured  in  the  design  space, 
and  have  the  ability  to  map  the  design  space  where  factors  can  be  reliably 
tested.  Further,  the  article  investigates  several  two-sided  approximate 
tolerance  factors  estimated  by  Monte  Carlo  simulation  and  compares 
them  to  the  exact  method.  Finally,  the  applicability  of  tolerance  intervals 
to  the  defense  T&E  community  is  presented  using  a  simple  case  study. 
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A  Comparative  Analysis  of  the  Value  of  Technology  Readiness 
Assessments 

Reginald  U.  Bailey,  Thomas  A.  Mazzuchi,  Shahram  Sarkani,  and 
a  826  David  F.  Rico 

The  U.S.  Department  of  Defense  endorsed  and  later  mandated  the  use  of 
Technology  Readiness  Assessments  (TRA)  and  knowledge-based  practices 
in  the  early  2000s  for  use  as  a  tool  in  the  management  of  program  acquisi¬ 
tion  risk.  Unfortunately,  implementing  TRAs  can  be  costly,  especially  when 
programs  include  knowledge-based  practices  such  as  prototyping,  perfor¬ 
mance  specifications,  test  plans,  and  technology  maturity  plans.  What  is  the 
economic  impact  of  these  TRA  practices  on  the  past  and  present  acquisition 
performance  of  the  U.S.  Army,  Navy,  and  Air  Force?  The  conundrum  today 
is  that  no  commonly  accepted  approach  is  in  use  to  determine  the  economic 
value  of  TRAs.  This  article  provides  a  model  for  the  valuation  of  TRAs  in 
assessing  the  risk  of  technical  maturity. 

Where  Are  the  People?  The  Human  Viewpoint  Approach  for 
Architecting  and  Acquisition 
£>852  Holly  A.  H.  Handley  and  Beverly  G.  Knapp 

The  U.S.  Department  of  Defense  Architecture  Framework  (DoDAF)  provides 
a  standard  framework  for  transforming  systems  concepts  into  a  consistent 
set  of  products  containing  the  elements  and  relationships  required  to  repre¬ 
sent  a  complex  operational  system.  However,  without  a  human  perspective, 
the  current  DoDAF  does  not  account  for  the  human  performance  aspects 
needed  to  calculate  the  human  contribution  to  system  effectiveness  and 
cost.  The  Human  Viewpoint  gives  systems  engineers  additional  tools  to 
integrate  human  considerations  into  systems  development  by  facilitating 
identification  and  collection  of  human-focused  data.  It  provides  a  way  to 
include  Human  Systems  Integration  (HSI)  constructs  into  mainstream 
acquisition  and  systems  engineering  processes  by  promoting  early,  frequent 
coordination  of  analysis  efforts  by  both  the  systems  engineering  and  HSI 
communities. 
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From  the  Chairman 
and  Executive  Editor 


The  theme  for  this  edition  of  Defense  Acquisition 
Research  Journal  is  "...All  Others  Must  Bring  Data.”  It 
derives  from  the  famous  quote  by  American  management 
consultant  W.  Edwards  Deming,  "In  God  we  trust;  all  oth¬ 
ers  must  bring  data,”  displayed  outside  the  office  of  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  Frank  Kendall.  Accurate  and  meaningful  data  are  the  basis 
for  making  informed  acquisition  policy  decisions;  as  British  scientist 
William  Thomson  (Lord  Kelvin)  said  over  a  century  ago,  “When  you  can 
measure  what  you  are  speaking  about ...  you  know  something  about  it;  but 
when  you  cannot  express  it  in  numbers,  your  knowledge  is  of  a  meagre 
and  unsatisfactory  kind."* 

The  first  article,  “Compressing  Test  and  Evaluation  by  Using  Flow 
Data  for  Scalable  Network  Traffic  Analysis,”  by  Kevin  Buell,  Mustafa  G. 
Baydogan,  Burhan  Senturk,  and  James  P.  Kerr,  examines  one  method  for 
accelerating  the  test  and  evaluation  phase  for  data  network-based  pro¬ 
grams  by  using  readily  scalable  flow  data.  The  second  article,  “Improving 
Statistical  Rigor  in  Defense  Test  and  Evaluation:  Use  of  Tolerance 
Intervals  in  Designed  Experiments,”  by  Alethea  Rucker,  looks  at  statisti¬ 
cal  tolerance  intervals  in  designed  experiments  as  an  analysis  method 
for  the  Department  of  Defense  test  and  evaluation  technical  community. 

In  “A  Comparative  Analysis  of  the  Value  of  Technology  Readiness 
Assessments,”  Reginald  U.  Bailey,  Thomas  A.  Mazzuchi,  Shahram 
Sarkani,  and  David  F.  Rico  provide  a  comparative  analysis  and  model 
of  the  economic  value  of  Technology  Readiness  Assessments  for  major 
defense  acquisition  programs.  Finally,  Holly  A.  H.  Handley  and  Beverly  G. 
Knapp  ask  “Where  Are  the  People?  The  Human  Viewpoint  Approach  for 
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Architecting  and  Acquisition,”  arguing  that  the  Department  of  Defense 
Architecture  Framework  needs  a  more  robust  link  between  the  system 
architecting  and  the  Human  Systems  Integration  communities. 

The  featured  book  in  this  issue’s  Defense  Acquisition  Professional 
Reading  List  is  Volume  II  in  the  History  of  Acquisition  in  the  Department 
of  Defense  series,  Adapting  to  Flexible  Response,  1960-1968,  by  Walter  S. 
Poole.  The  reviewer  is  Dr.  Roy  L.  Wood,  dean  of  DAU's  Defense  Systems 
Management  College. 

Finally,  I  encourage  prospective  authors  to  consider  submitting 
their  manuscripts  for  the  DAU  Alumni  Association’s  2015  Acquisition 
Symposium,  following  the  guidelines  in  the  Call  for  Papers  in  this  issue. 


< 


Dr.  Larrie  D.  Ferreiro 
Executive  Editor 
Defense  ARJ 


*  Thomson,  W.  (1889).  Lecture  on  electrical  units  of  measurement.  Popular 
Lectures  (Vol.  I).  London:  MacMillan. 

The  full  quote  is: 

I  often  say  that  when  you  can  measure  what  you  are  speaking  about,  and 
express  it  in  numbers,  you  know  something  about  it;  but  when  you  cannot 
express  it  in  numbers,  your  knowledge  is  of  a  meagre  and  unsatisfac¬ 
tory  kind;  it  may  be  the  beginning  of  knowledge,  but  you  have  scarcely, 
in  your  thoughts,  advanced  to  the  stage  of  science,  whatever  the  matter 
maybe,  (p.  73) 


DAU  ALUMNI  ASSOCIATION 

2015  HIRSCH 
RESEARCH  PAPER 
COMPETITION 


CALL  FOR  PAPERS 

Research  topics  may  include: 

•  Improve  Professionalism  of  the  Total  Acquisition  Workforce 

•  Career  Path  and  Incentives 

•  Agile  Program  Management 

•  Incorporating  Foreign  Military  Sales  and  Direct  Contractor  Sales  Strategies 
into  Programs 

•  Controlling  Costs  Throughout  the  Product  Life  Cycle 

•  System  Cyber  Hardness 


GROUND  RULES 

•  The  competition  is  open  to  anyone 
interested  in  the  DoD  acquisition 
system  and  is  not  limited  to 
government  or  contractor  personnel. 

•  Employees  of  the  federal  government 
(including  military  personnel)  are 
encouraged  to  compete  and  are 
eligible  for  cash  awards  unless  the 
paper  was  researched  or  written  as 
part  of  the  employee’s  official  duties 
or  was  done  on  government  time. 

If  the  research  effort  is  performed 
as  part  of  official  duties  or  on 
government  time,  the  employee 
is  eligible  for  a  noncash  prize,  i.e., 
certificate  and  donation  of  cash  prize 
to  a  Combined  Federal  Campaign 
registered  charity  of  winner’s  choice. 

•  First  prize  is  $1,000.  Second  prize 
is  $500. 


•  The  format  of  the  paper  must  be 
in  accordance  with  guidelines  for 
articles  submitted  for  the  Defense 
Acquisition  Research  Journal. 

•  Papers  are  to  be  submitted  to  the 
DAU  Director  of  Research: 
research@dau.mil. 

•  Papers  will  be  evaluated  by  a  panel 
selected  by  the  DAUAA  Board  of 
Directors  and  the  DAU  Director 
ofResearch. 

•  Award  winners  will  present  their 
papers  at  the  DAU  Acquisition 
Community  Training  Symposium, 
Tuesday,  April  7, 2015,  at  the  DAU 
Fort  Belvoir  campus. 

•  Papers  must  be  submitted  by 
December  16, 2014,  and  awards  will 
be  announced  in  January  2015. 
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DAU  Center  for 
Defense  Acquisition 
Research 

Research  Agenda  2015-2016 


The  Defense  Acquisition  Research  Agenda  is  intended  to  make 
researchers  aware  of  the  topics  that  are,  or  should  be,  of  particular 
concern  to  the  broader  defense  acquisition  community  throughout 
the  government,  academic,  and  industrial  sectors.  The  purpose  of 
conducting  research  in  these  areas  is  to  provide  solid,  empirically  based 
findings  to  create  a  broad  body  of  knowledge  that  can  inform  the  devel¬ 
opment  of  policies,  procedures,  and  processes  in  defense  acquisition, 
and  to  help  shape  the  thought  leadership  for  the  acquisition  community. 


Each  issue  of  the  Defense  ARJ  will  include  a  different  selection 
of  research  topics  from  the  overall  agenda,  which  is  at: 
http://www.dau.mil/research/Pages/researchareas.aspx 


Measuring  the  Effects  of  Competition 

•  What  means  are  there  (or  can  be  developed)  to  measure  the  effect 
on  defense  acquisition  costs  of  maintaining  an  industrial  base  in 
various  sectors? 

•  What  means  exist  (or  can  be  developed)  of  measuring  the  effect  of 
utilizing  defense  industrial  infrastructure  for  commercial  manu¬ 
facture  in  growth  industries?  In  other  words,  can  we  measure  the 
effect  of  using  defense  manufacturing  to  expand  the  buyer  base? 

•  What  means  exist  (or  can  be  developed)  to  determine  the  degree  of 
openness  that  exists  in  competitive  awards? 

•  What  are  the  different  effects  of  the  two  best -value  source-selec¬ 
tion  processes  (tradeoff  vs.  lowest  price  technically  acceptable)  on 
program  cost,  schedule,  and  performance? 


Strategic  Competition 
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Is  there  evidence  that  competition  between  system  portfolios  is  an 
effective  means  of  controlling  price  and  costs? 

Does  lack  of  competition  automatically  mean  higher  prices?  For 
example,  is  there  evidence  that  sole  source  can  result  in  lower 
overall  administrative  costs  at  both  the  government  and  industry 
levels,  to  the  effect  of  lowering  total  costs? 

What  are  the  long-term  historical  trends  for  competition  guidance 
and  practice  in  defense  acquisition  policies  and  practices? 

To  what  extent  are  contracts  being  awarded  noncompetitively  by 
congressional  mandate,  for  policy  interest  reasons?  What  is  the 
effect  on  contract  price  and  performance? 

What  means  are  there  (or  can  be  developed)  to  determine  the  degree 
to  which  competitive  program  costs  are  negatively  affected 
by  laws  and  regulations  such  as  the  Berry  Amendment  and  Buy 
America  Acts? 
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The  specialized  nature  of  technology-based  programs 
creates  volumes  of  data  on  a  magnitude  never  before 
seen,  complicating  the  test  and  evaluation  phase  of 
acquisition.  This  article  provides  a  practical  solution  for 
reducing  network  traffic  analysis  data  while  expediting 
test  and  evaluation.  From  small  lab  testing  to  full  integra¬ 
tion  test  events,  quality  of  service  and  other  key  metrics 
of  military  systems  and  networks  are  evaluated.  Network 
data  captured  in  standard  flow  formats  enable  scalable 
approaches  for  producing  network  traffic  analyses. 
Because  of  its  compact  representation  of  network  traffic, 
flow  data  naturally  scale  well.  Some  analyses  require  deep 
packet  inspection,  but  many  can  be  calculated/approxi¬ 
mated  quickly  with  flow  data,  including  quality-of-service 
metrics  like  completion  rate  and  speed  of  service. 


«  Image  designed  by  Diane  Fleischer 
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Challenges  for  Acquisition 

With  two  major  conflicts  coming  to  an  end— Iraq  in  late  2011  and  the 
expected  end  of  U.S.-led  combat  operations  this  year  in  Afghanistan— it 
comes  as  little  surprise  that  budgets  throughout  the  Department  of 
Defense  are  entering  an  age  of  austerity.  The  earlier  enacted  across- 
the-board  federal  spending  cuts,  known  as  sequestration,  claimed  a 
percentage  of  the  Defense  Department’s  budget.  Despite  these  bud¬ 
get  cuts,  the  expectation  that  defense  acquisition  professionals  will 
field  technology-based  systems  to  the  warfighter  is  at  an  all-time  high. 
Low-intensity  conflict  operations  throughout  the  world  rely  heavily  on 
technology  and  intelligence  systems. 

Complicating  the  acquisition  of  these  technology-based  systems  and 
programs  is  the  voluminous  amounts  of  data  they  produce,  as  observed  at 
the  Army’s  recurring  large  technology  test  event,  the  Network  Integrated 
Evaluation  (NIE).  NIE  produces  terabytes  of  network  data  in  a  single 
day;  this  amount  of  data  is  simply  too  large  to  manage  and  far  too  large 
for  test  and  evaluation  (T&E)  professionals  to  efficiently  analyze  the 
data.  Processing  a  single  data  set  can  take  as  long  as  24  to  36  hours;  the 
status  quo  is  grossly  inefficient  to  meet  the  needs  of  rapid  acquisition. 

Because  of  these  inefficiencies,  meaningful  and  effective  engineer¬ 
ing  modifications  performed  during  a  test  event  cannot  be  done  fast 
enough.  The  delay  between  analysis,  engineering  modifications  based 
on  data,  and  validation  can  extend  the  test  event— or  worse,  neces¬ 
sitate  a  follow-on  event.  Both  scenarios  require  a  longer  T&E  phase 
whether  in  developmental  test,  operational  test,  or  integrated  test,  which 
impacts  schedule  and  cost  by  extending  the  T&E  phase  of  the  acquisi¬ 
tion  life  cycle.  Moving  programs  that  involve  complex  information  and 
communication  networks  from  the  Technology  Development  Phase 
(Milestone  B)  to  the  Engineering  &  Manufacturing  Development  phase 
of  the  Integrated  Defense  Acquisition,  Technology  and  Logistics  Life 
Cycle  Management  System  (Cochrane  &  Brown,  2010)  is  notoriously  dif¬ 
ficult  for  a  variety  of  reasons,  not  the  least  of  which  is  inefficient  methods 
of  handling  T&E  data. 

Engineers  from  the  U.S.  Army  Electronic  Proving  Ground,  with  their 
academic  partners  at  Arizona  State  University’s  Security  and  Defense 
Systems  Initiative,  developed  a  way  to  compress  some  of  the  T&E  time¬ 
line  for  defense  technology  systems  that  use  networks.  This  partnership 
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realized  nominal  gains  of  75  percent,  reducing  analysis  time  from  24 
hours  to  as  little  as  six  for  data  sets  of  approximately  two  terabytes.  The 
efficiency  gains  are  a  sum  of  statistical  and  probabilistic  modeling,  data 
reduction,  and  the  use  of  commercial-off-the-shelf  software  (COTS) 
for  analyzing  network  traffic.  These  gains  are  a  partial  answer  to  the 
challenge  introduced  at  the  beginning  of  this  article:  that  defense  acqui¬ 
sition  professionals  must  manage  to  field  capable  technology  systems  to 
the  warfighter. 


Background 

Analysis  of  network  traffic  has  been  studied  for  some  time,  and 
a  well-established  body  of  research  exists  on  Internet  measurement 
(Crovella  &  Krishnamurthy,  2006).  However,  evaluation  of  networks 
within  military  test  exercises  has  some  unique  characteristics  not 
shared  with  general  Internet  measurement.  For  example,  while  Internet 
measurement  is  largely  focused  on  collecting  data  at  routers,  military 
test  exercises  often  focus  more  on  collecting  data  at  end-nodes.  In  some 
ways,  this  is  a  luxury  enabled  by  the  contained  evaluation  environment, 
which  allows  for  accurate  point-to-point  metrics  like  speed  of  service. 

On  the  other  hand,  collecting  data  at  end-nodes  can  account  only 
for  traffic  seen  on  those  nodes.  The  network  may  not  be  large  enough  for 
router-based  measurements  to  be  useful,  so  end-node  collection  may 
be  the  only  option.  This  unique  environment  is  not  well  explored  in  the 
current  literature  on  Internet  measurement. 

Military  test  exercises  present  further  challenges.  Because  of  the 
nature  of  field  exercises,  harvesting  data  may  not  occur  as  frequently 
as  test  officers  would  like.  Interfaces  and  protocols  may  not  be  in  place 
to  gather  test  data,  and  it  often  must  be  copied  and  physically  carried 
from  nodes  under  test.  These  unique  aspects  can  lead  to  data  collection 
inconsistencies  and  errors. 

Nevertheless,  providing  access  to  measurement  results  is  essential 
for  providing  information  that  may  affect  ongoing  testing.  F or  example, 
determining  whether  a  data  collection  system  is  working,  whether  a 
node  is  active,  and  how  much  data  has  been  collected  are  all  important 
to  know  as  soon  as  possible  during  testing. 
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As  the  amount  of  data  flowing  over  networks  increases,  the  ability 
to  collect,  transfer,  store,  process,  and  analyze  the  data  becomes  more 
challenging.  Many  tools  and  standards  from  the  Internet  measurement 
community  can  be  useful  here.  Where  possible,  using  open-source  soft¬ 
ware  is  preferable,  thereby  decreasing  costs  and  avoiding  “reinventing 
the  wheel.”  Using  standard  data  formats  is  pivotal  to  knowledge  transfer 
and  tool  interoperability.  When  compared  to  solutions  developed  in- 
house,  the  open  source  solution  often  offers  greater  performance,  and  a 
larger  support  community,  and  it  is  typically  far  less  costly. 


As  the  amount  of  data  flowing  over  networks 
increases,  the  ability  to  collect,  transfer,  store, 
process,  and  analyze  the  data  becomes  more 
challenging. 


Traffic  Data  Formats 

To  record  network  traffic,  two  main  formats  are  generally  available: 
packet  capture  and  flow.  Packet  capture  formats  record  everything  that 
goes  across  the  wire  (each  individual  packet,  including  header  and  data). 
Flow  formats  summarize  the  traffic  and  exclude  the  content.  Excluding 
the  content  sacrifices  the  ability  to  reconstruct  true  network  traffic. 
However,  focusing  on  flow  formats  allows  for  important  metrics  about 
the  network  traffic— like  performance,  quality  of  service,  and  loss— but 
alleviates  the  burden  of  fully  constituted  packets. 

Flow  formats  are  attractive  because  working  with  large  sizes  of  data 
presents  bottlenecks,  particularly  with  disk  reads  and  writes.  Smaller 
data  sizes  result  in  faster  analysis  performance.  The  most  popular 
flow  format  has  been  NetFlow  from  Cisco,  but  it  has  been  standard¬ 
ized  into  a  nonproprietary  flow  format  called  Internet  Protocol  (IP) 
Flow  Information  Export,  or  IPFIX  (Internet  Engineering  Task  Force, 
n.d.),  and  several  other  flow  formats  are  precursors  or  variations  on  the 
flow  concept. 
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Flows  normally  summarize  traffic  by  recording  the  number  of  pack¬ 
ets,  total  bytes,  flags,  protocols,  and  other  elements  over  some  time  period 
(e.g.,  1  minute)  from  a  source  IP  address  to  a  destination  IP  address. 
These  are  referred  to  as  aggregated  flows.  An  alternative  is  to  produce 
one  flow  per  packet,  referred  to  as  single-packet  flows.  These  enable  some 
quality-of-service  and  other  advanced  analyses  discussed  later. 

Some  tools  also  work  natively  with  flow  data  that  have  been  com¬ 
pressed  using  standard  compression  algorithms.  Table  1  represents  a 
sample  calculation  based  on  various  observations  when  working  with 
flow  data.  Exact  numbers  will  vary  based  on  traffic  characteristics,  but 
this  gives  some  idea  of  the  significant  data  reduction  achieved  when  using 
flow  data,  which  in  turn  eliminates  the  disk  throughput  bottleneck. 


TABLE  1.  SAMPLE  FLOW  SIZES  FOR  A  GIVEN  SET  OF  PACKET 
CAPTURES 


Packet 

Capture 

Single- 

Packet 

Flow 

Single- 
Packet  Flow 

(Compressed) 

Aggregated 

Flow 

Aggregated 

Flow 

(compressed) 

Size  500  GB 

38  GB 

2  GB 

0.25  GB 

0.04  GB 

Data  Preparation 

Network  traffic  data  must  be  collected  and  prepared  for  analysis  to 
support  real-time  queries  and  in-depth  discovery.  Ideally,  traffic  data 
are  collected  natively  in  flow  format  (such  capabilities  are  built  into 
most  routers).  Since  some  applications  require  full  packet  contents, 
other  approaches  may  be  required,  such  as  capturing  full  packet  data 
and  producing  flows  from  it  or  capturing  both  full  packet  and  flow  data 
simultaneously. 

Reading  packet  capture  data  is  normally  bounded  by  disk  I/O  (input/ 
output),  but  writing  it  to  the  significantly  smaller  flow  format  is  gener¬ 
ally  not.  Optimized  open  source  tools  like  Yet  Another  Framework,  or 
YAF  (Software  Engineering  Institute,  2006),  convert  standard  Libpcap- 
formatted  packet  captures  (Libpcap  is  a  portable  library  for  network 
traffic  capture)  to  IPFIX  format.  Conversion  time  is  largely  a  product 
of  disk  read  speed.  For  example,  on  a  current  commodity  system  with 
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lOOMB/s  read  speed,  conversions  performed  from  packet  capture  to  flow 
took  about  11  seconds/GB.  This  means  that  converting  500GB  of  packet 
capture  would  take  about  1.5  hours. 

Generally,  two  approaches  are  used  to  produce  a  final  set  of  flow  data. 
For  analyses  that  do  not  compare  traffic  between  two  traffic  collection 
points  (e.g.,  traffic  load,  protocol  distribution,  topology),  the  following 
steps  are  required: 

1.  Convert  all  packet  capture  files  to  aggregated  flow  files. 

2.  Combine  the  aggregated  flow  files  into  a  single  aggregated  flow 
file. 

3.  Deduplicate  the  single  aggregated  flow  file. 

The  final  flow  file  is  deduplicated  to  avoid  double  counting  traffic 
seen  at  a  source  and  destination  traffic  collection  point.  An  accurate  (but 
approximated)  deduplication  process  is  available  in  some  flow-based 
tools  and  is  accomplished  by  matching  flows  based  on  time,  protocol, 
bytes,  and  so  on  within  a  configurable  threshold. 

For  analyses  based  on  matching  packets  between  traffic  collection 
points  (e.g.,  completion  rate,  speed  of  service),  the  following  steps  are 
required: 

1.  Convert  all  packet  capture  files  to  single  packet  flow  files. 

2.  Combine  the  single  packet  flow  files  producing  one  flow  file  per 
traffic  collection  point. 

The  resulting  files  are  not  deduplicated  to  enable  matching  of  traffic 
between  files  for  these  types  of  analyses. 

Traffic  Analysis 

An  extensive  search  and  evaluation  of  open  source  network  traffic 
analysis  tools  yields  several  that  are  particularly  noteworthy  and  use¬ 
ful.  Of  course,  Wireshark  and  tshark  are  popular  tools  providing  packet 
inspection  capabilities  (Wireshark,  n.d.),  but  their  performance  in  many 
respects  is  lacking,  particularly  for  processing  many  large  files.  A  library 
called  libtrace  provides  optimal  results  for  working  with  packet  capture 
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files  (WAND  Network  Research  Group,  n.d.).  Argus  is  another  open 
source  tool  that  provides  significant  functionality  as  well  as  a  proprietary 
flow  format  that  includes  some  additional  information  that  could  be  use¬ 
ful  for  characterizing  traffic  (QoSient,  2014). 

One  noteworthy  tool  that  is  well-supported,  highly  optimized,  and 
contains  all  the  basic  functionality  one  would  expect  for  data  prepara¬ 
tion  and  traffic  analysis  is  SiLK  (Software  Engineering  Institute,  2006). 
Traffic  analyses  enabled  by  aggregated  flow  files  using  SiLK  queries 
include  (but  are  not  limited  to): 

•  Topology  by  generating  lists  of  source/destination  pairs; 

•  Finding  all  traffic  communicating  on  a  given  port; 

•  Separating  nodes  or  pairs  into  bins  based  on  percentage  of 
total  load;  and 

•  Configurable  filtering  and  traffic  identification  based  on 
any  combination  of  port,  protocol,  IP  address,  flow  start/ 
end/duration,  etc. 


Defense  ARJ,  October  2014,  Vol.  21  No.  4: 788-802 


794 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Visualization  is  not  the  focus  of  this  article,  but  two  open  source 
tools  should  be  mentioned  here.  The  Ozone  Widget  Framework  (Next 
Century,  n.d.)  has  proven  to  be  very  useful.  Also,  an  extensive,  clean,  and 
optimized  JavaScript  library  for  visualizing  many  types  of  data  can  be 
found  in  D3-Data  Driven  Documents  (Bostock,  2013). 

Quality  of  Service  from  Flow 

Two  essential  metrics  of  network  traffic  are  delay  and  loss.  Delay  is 
sometimes  referred  to  as  speed  of  service  and  loss  is  sometimes  measured 
by  completion  rate.  Given  two  files  of  single-packet  flows  representing 
two  traffic  collection  points— each  of  which  is  associated  with  one  or 
more  nodes— the  traffic  loss  between  the  two  is  determined  by  simply 
adding  the  number  of  packets  in  the  file  representing  the  destination, 
and  then  subtracting  the  number  of  packets  in  the  source. 

Calculating  delay  is  generally  more  complicated  and  subject  to  some 
error  with  flow  data.  Using  files  from  two  traffic  collection  points,  packets 
can  be  matched  based  on  characteristics  like  size  and  protocol,  as  well 
as  timestamp  within  a  given  threshold  as  shown  in  Figure  1.  The  time- 
stamps  for  a  given  packet  will  be  different  in  the  source  and  destination 
precisely  because  of  delay. 


FIGURE  1.  DELAY  AND  LOSS  BETWEEN  SENDER  AND  RECEIVER 
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For  a  given  source  IP  and  destination  IP  pair,  reading  all  single  packet 
flows  into  memory  and  matching  packets  is  feasible.  To  calculate  the 
speed  of  service,  simply  subtract  the  timestamp  of  the  matched  packet 
in  the  source  from  that  in  the  destination,  and  then  average  the  differ¬ 
ence  over  all  the  matched  pairs.  This  is  not  matching  packets  based  on 
content,  and  working  with  a  timestamp  threshold  for  matching  will  pro¬ 
duce  some  error.  However,  packet  matching  with  only  flow  data— when 
compared  with  matching  using  full  packet  content— provides  nearly 
identical  results. 


Two  essential  metrics  of  network  traffic  are  delay 
and  loss. 


In  fact,  the  average  delay  over  some  number  of  packets  (versus  delay 
for  an  individual  packet)  can  be  calculated  accurately  using  a  number  of 
approaches.  Though  lost  packets  must  be  accounted  for,  matching  each 
packet  precisely  is  not  essential  to  produce  an  accurate  average.  Instead, 
ensure  that  packets  are  simply  matched  (uniquely)  with  some  nearby 
packet  in  the  source  and  destination.  Mathematically,  this  is  because 
(bl  -  al)  +  (b2  -  a2)  is  identical  to  (bl  -  a2)  +  (b2  -  al).  In  fact,  and  if  it 
appears  more  computationally  feasible,  an  alternative  is  to  add  all  time- 
stamps  over  a  given  period  from  the  destination,  and  then  subtract  the 
corresponding  timestamps  from  the  source,  i.e.,  (bl  +  b2)  -  (al  +  a2).  After 
dividing  by  the  number  of  packets,  the  resultant  average  delay  is  the  same 
as  exact  pairwise  matching. 

Since  flow  formats  generally  provide  for  some  extensibility,  one 
approach  to  improve  matching  accuracy  is  to  compute  a  reasonable, 
locally  unique  hash  value  per  packet  based  on  its  contents.  This  hash 
value  is  stored  as  additional  information  within  the  flow  record.  Indeed, 
flow  extensibility  can  be  an  important  means  of  bridging  the  gap  between 
full  packet  capture  and  flow  data  by  allowing  for  small,  but  critical  pieces 
of  data  from  some  packets  to  be  stored  with  flow  records. 

Future  Work 

Some  analyses,  which  are  now  only  accomplished  using  packet  cap¬ 
ture  data  could  be  accomplished  using  flow  data,  but  may  require  some 
statistical  analysis  or  algorithmic  development.  For  example,  reporting 
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quality  of  service  based  on  message  type  (e.g.,  voice  versus  video  versus 
Web  document)  might  normally  require  deep  packet  inspection.  However, 
a  flow  format  could  save  an  indication  of  message  type  when  converting 
from  packet  capture  to  flow,  using  extensibility.  Also,  traffic  application 
mining  has  been  studied  extensively  and  could  be  used  on  flows  to  add 
some  information  about  message/application  type. 

Many  statistical  approaches  are  available  that  could  provide  value 
in  this  domain.  Finding  cause/effect  of  poor  network  conditions,  find¬ 
ing  anomalies,  and  other  problems  could  be  solved  with  statistical  data 
mining  approaches.  For  example,  low  quality  of  service  may  be  caused  by 
many  factors  including  high  traffic  volume  (and  associated  congestion), 
proximity  of  sender  and  receiver,  or  physical  conditions  such  as  obstacles 
in  the  path  between  sender  and  receiver. 

Advanced  analyses  may  also  highlight  where  significant  events  occur 
and  which  nodes  are  involved  in  these  events.  They  may  include  ways  to 
group  and  visualize  nodes  beyond  standard  clustering  like  logical  topol¬ 
ogy  and  geographical  display.  Node  groups  may  instead  be  formed  based 
on  traffic  profile  matching  and/or  quality-of-service  similarities. 

Conclusions 

As  the  volume  of  network  traffic  data  increases,  analysis  of  military 
systems  and  networks  becomes  more  challenging.  Flow  data  have  been 
used  by  the  general  Internet  measurement  community  for  some  time  to 
enable  scalable  traffic  analysis  for  cyber  security  and  traffic  engineering. 
However,  flow  data  have  not  been  widely  applied  to  metrics  requiring 
more  precision  and  advanced  analytics.  Military  exercises  are  unique  in 
that  they  are  generally  a  smaller,  contained  set  of  traffic,  often  utilizing 
end-to-end  measurements. 

Flow  data  are  much  smaller  than  full  packet  captures  and  thus 
address  the  I/O  bottleneck  common  in  data  processing  for  network 
measurement.  Compressed  flows  provide  even  more  data  minimization. 
Common  flow  formats  allow  for  some  extensibility  so  that  essential 
pieces  of  payload  can  be  kept  within  a  flow  when  needed. 

Traffic  flow  data  directly  enable  basic  traffic  characterizations  like 
load  and  protocol  distribution,  and  these  are  handled  well  by  open  source 
tools  that  work  with  flow  data.  More  advanced  analyses  like  quality  of 
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service  are  also  possible  using  flow  data  and  flow  tools  with  additional 
algorithmic  support.  By  capturing  flows  composed  of  a  single  packet  and 
matching  these  flows  at  source  and  destination,  delay  and  packet  loss  can 
be  calculated  accurately  using  flow  data. 

Statistical  approaches  could  be  used  for  even  more  high-level  traf¬ 
fic  characterization  such  as  cause/effect  analysis  of  network  traffic 
conditions.  Advanced  analyses  may  also  highlight  where  significant 
events  occur  and  which  nodes  are  involved  in  these  events.  Grouping 
nodes  based  not  only  on  logical  topology  and  physical  location,  but  also 
traffic  pattern  similarity  would  also  provide  additional  understanding 
and  enhanced  visual  analytics.  These  provide  greater  insight  into  net¬ 
work  traffic  data  that  can  point  operators  to  potential  areas  for  further 
analysis. 

These  approaches  to  increase  the  efficiency  of  network  traffic  analy¬ 
sis  are  small,  but  indicative  of  the  trend  to  meet  the  big  data  problem. 
The  length  of  T&E  for  technology-based  programs  will  continue  to  grow 
as  the  complexity  and  interdependencies  of  these  systems  grow.  The 
challenge  of  big  data  can  be  mitigated  through  incremental  improve¬ 
ments.  Using  sensible,  pragmatic  methods  that  reduce  the  data— through 
statistical  and  probabilistic  modeling  coupled  with  the  acceleration  of 
analysis  by  adopting  COTS— is  one  way  to  manage  this  big  data  challenge. 
Regardless  of  the  challenge,  defense  acquisition  professionals  must  look 
for  new  ways  to  enable  our  increasingly  technology-enhanced  warfighter. 
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Improving  Statistical 
Rigor  in  Defense  Test 
and  Evaluation: 

Use  of  Tolerance  Intervals  in 
Designed  Experiments 


Alethea  Rucker 


Leveraging  the  use  of  statistical  methods  is  critical  in 
providing  defensible  test  data  to  the  Department  of 
Defense  Test  and  Evaluation  (T&E)  enterprise.  This  article 
investigates  statistical  tolerance  intervals  in  designed 
experiments  for  the  T&E  technical  community.  Tolerance 
intervals  are  scarcely  discussed  in  extant  literature  as 
compared  to  confidence/prediction  intervals.  The  lesser 
known  tolerance  intervals  can  ensure  a  proportion  of  the 
population  is  captured  in  the  design  space,  and  have  the 
ability  to  map  the  design  space  where  factors  can  be 
reliably  tested.  Further,  the  article  investigates  several 
two-sided  approximate  tolerance  factors  estimated  by 
Monte  Carlo  simulation  and  compares  them  to  the  exact 
method.  Finally,  the  applicability  of  tolerance  intervals  to 
the  defense  T&E  community  is  presented  using  a  simple 
case  study. 
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In  the  FY  2012  Annual  Report  from  the  Director,  Operational  Test 
and  Evaluation  (DOT&E),  the  director  identified  two  areas  as  requiring 
further  improvement  to  move  toward  institutionalizing  statistical  rigor: 

(a)  “execution  of  testing  in  accordance  with  the  planned  test  design”  and 

(b)  “analysis  of  test  data  using  advanced  statistical  methods  commen¬ 
surate  with  test  designs  developed  using  DOE  [Design  of  Experiments]” 
(Gilmore,  2012a,  p.  v).  The  report  further  states  that  current  data  analysis 
is  “limited  to  reporting  a  single  average  (mean)  of  the  performance  across 
all  the  test  conditions”  (p.  v).  In  doing  so,  efficiencies  achieved  through 
meticulous  test  planning  and  design  are  discarded.  Realizing  the  need 
for  increased  rigor,  a  Defense  Science  of  Test  Research  Consortium 
was  formed  in  2011,  partnered  with  Arizona  State  University,  Virginia 
Polytechnic  Institute  and  State  University,  Naval  Postgraduate  School, 
and  the  Air  Force  Institute  of  Technology  (AFIT).  The  consortium’s 
overall  research  goal  is  to  support  the  incorporation  of  advanced  statisti¬ 
cal  rigor  and  mathematical  foundations  into  the  test  enterprise  (AFIT, 
2012).  Research  largely  focuses  on  improved  experimental  design  and 
statistical  theory  (Freeman,  Ryan,  Kensler,  Dickinson  &  Vining,  2013; 
Haase,  Hill,  &  Hodson,  2011;  Hill,  Gutman,  Chambal,  &  Kitchen,  2013; 
Johnson,  Hutto,  Simpson,  &  Montgomery,  2012).  The  research  of  toler¬ 
ance  intervals  in  designed  experiments  has  yet  to  be  fully  discussed.  This 
article  continues  the  research  dialogue  and  adds  to  the  body  of  knowl¬ 
edge  of  tolerance  interval  literature  in  defense  testing,  particularly  the 
Scientific  Test  and  Analysis  Techniques  (STAT)  implementation  effort. 
Further,  this  article  aims  to  assist  primarily  test  and  evaluation  (T&E) 
practitioners  such  as  engineers,  analysts,  and  test  project/program  man¬ 
agers  in  understanding  how  the  use  of  statistics  can  greatly  improve  the 
quality  of  results  in  the  decision-making  process  and  improve  credibility 
through  objective  data. 


Purpose 

The  purpose  of  this  article  is  twofold.  First,  tolerance  intervals 
are  rarely  discussed  in  extant  literature  as  having  application  to  the 
defense  T&E  community.  This  research  closes  the  gap  by  exploring  the 
applicability  of  tolerance  intervals  in  designed  experiments.  An  attempt 
to  use  tolerance  intervals  in  defense  testing  was  investigated  by  the 
National  Research  Council  (NRC)  of  the  National  Academies  on  test¬ 
ing  body  armor  materials.  Their  recently  published  work  (NRC,  2012) 
recommended  use  of  statistical  tolerance  bounds,  but  their  examples 
were  confined  to  single,  normally  distributed  samples,  and  did  not  take 


805 


Defense  ARJ,  October  2014,  Vol.  21  No.  4:804-824 


Improving  Statistical  Rigor  in  Defense  Test  and  Evaluation:  Use  of  Tolerance  Intervals  in  Designed  Experiments 


into  account  the  design  structure.  Second,  increased  statistical  rigor  is 
needed  in  defense  testing  and  analysis  due  to  the  complexity  and  chal¬ 
lenges  in  testing  a  defense  weapon  system.  Recently,  the  use  of  STAT  has 
gained  traction  within  the  Department  of  Defense  (DoD)  T&E  commu¬ 
nity  (DoD,  2012;  Gilmore,  2010;  Operational  Test  Agencies,  2009).  Albeit 
gradual,  the  defense  community  is  leveraging  the  long-spanning,  rich  his¬ 
tory  of  statistical  methods  in  industry  and  replacing  the  budget-driven 
test  events,  combat  scenarios,  and  one-factor-at-a-time  approach  with  a 
statisticaly  rigorous  approach  to  test  design  using  DOE  (Johnson  et  al., 
2012).  Though  current  guidance  and  emphasis  on  the  use  of  designed 
experiments  in  test  plans  is  sufficient  in  explaining  test  planning  and 
design,  it  falls  short  of  providing  specific  guidance  on  test  analysis  and 
reporting  to  the  decision  makers,  who  ultimately  decide  on  whether  to 
field  the  weapon  system  to  the  warfighter.  In  these  resource-constrained 
times,  providing  the  T&E  community  defensible  and  objective  test  data 
to  enable  risk  management  for  leadership  during  system  development, 
procurement,  and  operation  is  imperative. 


Increased  statistical  rigor  is  needed  in  defense 
testing  and  analysis  due  to  the  complexity  and 
challenges  in  testing  a  defense  weapon  system. 


What  Are  Tolerance  Intervals? 

The  importance  of  tolerance  intervals  has  long  been  recognized 
(Wilks,  1941, 1942),  with  wide  applicability  to  areas  such  as  manufactur¬ 
ing,  pharmaceutical,  quality  control,  engineering,  and  material  science 
commonly  referred  to  as  A-  and  B-basis  allowables.  In  general,  toler¬ 
ance  intervals  capture  a  fixed  proportion  of  population  (p)  with  a  given 
confidence  level  (1-a).  Confidence  intervals  are  the  most  commonly  used 
statistical  interval  method  focused  on  parameters  such  as  mean  and/ 
or  standard  deviation,  while  prediction  intervals  consider  the  predic¬ 
tion  of  individual  responses.  Prediction  intervals  are  useful  only  if  the 
sample  on  which  the  interval  is  based  represents  the  population,  but  if 
the  population  changes  over  time,  then  the  prediction  interval  is  useless 
(Vining,  1997).  In  other  practical  instances,  the  proportion  of  popula¬ 
tion,  rather  than  mean  is  of  interest,  rendering  tolerance  intervals  more 
appropriately  applied  in  those  situations.  A  tolerance  interval  allows 
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us  to  make  statements  surrounding  the  distribution  rather  than  the 
predicted  individual  responses,  such  as:  "We  are  95  percent  confident 
that  at  least  90  percent  of  the  population  distribution  will  lie  within  the 
specified  interval."  Unfortunately,  tolerance  intervals  are  the  least  dis¬ 
cussed  interval  in  extant  literature.  Jensen  (2009)  attributes  this  to  the 
difficulty  of  computation  and  lack  of  statistical  software  packages  that 
readily  offer  tolerance  intervals.  De  Gryze,  Langhans,  and  Vandebroek 
(2007)  indicated  practical  guidelines  to  calculate  and  use  tolerance 
intervals  in  real-world  applications  are  currently  absent  and  that  for  even 
the  simplest  regression  model,  tolerance  intervals  are  lacking. 

The  relevant  interval  in  some  situations  in  defense  testing  such  as 
body  armor  testing  should  be  one  that  states  a  specified  proportion  of 
population  that  falls  above  or  below  some  threshold  limit  versus  merely 
reporting  the  mean  of  the  response.  Many  researchers  have  conducted 
studies  in  the  construction  of  tolerance  limits  for  normal  distribution; 
early  works  by  Wilks  (1941, 1942),  Wald  (1943),  and  Wald  and  Wolfowitz 
(1946)  are  widely  available  in  the  literature.  Exact  methods  for  one-sided 
and  two-sided  tolerance  regions  have  been  researched  for  the  normal 
distribution.  Tolerance  intervals  for  linear  regression  models  were 
first  introduced  in  the  seminal  paper  by  Wallis  (1951).  Wallis  extended 
the  previous  work  of  Wald  and  Wolfowitz  (1946)  for  a  normally  distrib¬ 
uted  sample  to  a  linear  regression  model.  Since  then,  researchers  have 
extended  Wallis’s  work  to  multiple  and  multivariate  tolerance  intervals 
(Krishnamoorthy  &  Mondal,  2008;  Lee  &  Mathew,  2004).  The  contin¬ 
ued  research  in  this  field  has  allowed  T&E  practitioners  to  expand  their 
analysis  and  evaluation  to  include  multiple  responses  such  as  time  to 
acquire  a  target  and  miss  distance.  Not  discussed  in  this  article  are 
Bayesian  methods  that  incorporate  a  priori  information  and  are  useful 
in  rolling  up  and  including  developmental  test  data  and/or  subject  matter 
expert  opinions.  For  a  review  of  the  statistical  tolerance  region,  including 
Bayesian  tolerance  intervals,  see  Krishnamoorthy  and  Mathew  (2009). 

Design  of  Experiments  Framework 

A  simplified  DOE  framework  (Figure  1)  is  proposed  for  use  in  the  DoD 
T&E  community  and  later  applied  to  the  case  study.  Another  suggested 
framework  is  the  Plan,  Design,  Execute,  Analyze  model  developed  by 
Eglin  Air  Force  Base,  Florida. 
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FIGURE  1.  GENERALIZED  DOE  FRAMEWORK 


Phase  O:  Mission  Goal  Definition 

This  step  is  accomplished  at  the  top  level  and  early  in  the  acquisition 
cycle,  where  the  mission  statement  and  objectives  (Critical  Operational 
Issues,  or  COIs)  are  clearly  defined  for  the  program.  COIs  answer  the 
question,  “What  capability  will  the  system  provide?”  A  hierarchy  can 
serve  as  a  catalyst  for  generating  discussion  about  the  identification  of 
factors,  levels,  and  responses  for  the  proposed  tests.  This  is  accomplished 
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by  the  test  team— a  T&E  Working-level  Integrated  Program  Team 
(WIPT).  The  T&E  WIPT  membership  should  include  all  stakeholder 
organizations  from  the  developmental  test  and  operational  test  commu¬ 
nities.  Test  membership  should  include,  but  is  not  limited  to,  the  program 
manager,  operators,  subject  matter  experts,  program  analysts,  testers, 
and  requirements  representative.  Generation  of  goals,  objectives,  factors, 
levels,  and  responses  should  be  an  exhaustive  process  so  no  input  and 
output  variables  are  left  out.  Therefore,  continuously  including  these 
members  upfront  is  critical  to  improving  test  outcomes. 

Phase  1:  Plan  and  Design 

Test  goal  and  objective.  Every  good  experimental  design  begins 
with  a  clear,  concise  goal  and  objective  that  is  well  understood  by  all 
parties  before  test  planning.  The  right  kind  of  questions  leading  to 
development  of  quantifiable  terms  (responses)  need  to  be  articulated 
for  effective  test  execution  and  data  collection.  The  right  quantitative 
metrics  are  essential  for  developing  a  good  test  design;  poorly  chosen 
or  ill-defined  measures  can  lead  to  unnecessary  costs  or  ambiguous 
test  results  (Gilmore,  2012b).  Continuous  metrics,  such  as  detection 
range,  enable  the  most  efficient  use  of  resources  and  provide  the  most 
information.  On  the  other  hand,  binary  metrics,  such  as  pass  or  fail,  hit 
or  miss,  offer  less  information  to  testers  and  can  increase  test  resource 
requirements. 

Response  variables,  factors,  and  levels.  Selection  of  a  response 
variable,  continuous  or  discrete,  should  be  carefully  considered  to  mini¬ 
mize  risk  in  running  into  a  Type  I  (a)  or  Type  II  (/?)  error.  Responses  are 
Key  Performance  Parameters,  Measures  of  Effectiveness,  Measures 
of  Suitability,  Critical  Technical  Parameters,  Key  System  Attributes, 
and/or  Measures  of  Performance  that  are  documented  and  traced  to 
the  requirements  document  (Gilmore,  2012b).  In  current  DoD  test  plan¬ 
ning,  the  statistical  measures  of  merit— power  (1-/?)  and  confidence  level 
(1-a)— must  be  documented  in  the  Test  and  Evaluation  Master  Plan 
(TEMP;  see  Gilmore,  2012b).  The  Type  I  error  (a)  is  the  probability  of 
declaring  a  factor  is  affecting  the  response  when  in  reality  it  does  not. 
This  percent  value  is  typically  agreed  upon  by  the  decision  maker  based 
on  inputs  from  the  T&E  WIPT.  The  quantity  expresses  the  decision  mak¬ 
ers’  risk  tolerance  for  making  a  wrong  decision  based  on  limited  test  data 
(Freeman,  Glaeser,  &  Rucker,  2011).  A  Type  II  error  (//)  is  the  probability 
of  declaring  a  factor  does  not  affect  the  response,  when  in  reality  it  does. 
Power  (1-/?)  is  the  likelihood  of  not  making  the  /?  error  and  the  ability  to 
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detect  differences.  This  is  set  by  the  test  team  during  test  planning.  In 
general,  the  confidence  levels  are  set  between  80%  and  95%  (a  =  0.20 
to  0.05),  and  the  power  for  a  signal-to-noise  S:N  =  1.0  should  be  above 
80%  (Department  of  the  Army,  2012).  Both  types  of  errors  must  be  well 
understood  and  explained  to  the  decision  maker  due  to  the  unintended 
programmatic  consequences  that  might  result  from  a  lack  of  under¬ 
standing.  Defining  factors  (independent  variables)  is  no  trivial  matter 
and  must  be  determined  by  the  entire  test  team.  The  factors  define  the 
operational  environment  of  the  system.  Some  proven  effective  brain¬ 
storming  methods  to  aid  the  process  are  fishbone  diagrams  (also  known 
as  Ishikawa  Diagrams)  and  process  flow  diagrams.  Note  that  it  is  better 
to  include  more  factors  than  preclude  factors  that  might  be  significant 
(Telford,  2007).  Levels  are  the  specified  values  of  the  factors,  and  the 
general  recommendation  is  to  consider  two  to  three  levels  for  each  factor 
(Freeman  et  al.,  2011). 

Test  design.  The  test  design  is  constructed  after  the  factor,  levels, 
and  responses  are  identified  by  the  test  team.  Decision  trade-offs  between 
risk  and  costs  are  made  at  this  stage  with  assistance  from  the  test  team, 
especially  when  test  resources  are  limited.  The  choice  of  design  involves 
consideration  of  sample  size,  selection  of  a  suitable  run  order  for  the 
trials,  and  whether  blocking  or  other  randomization  restrictions  exist 
(Montgomery,  2001).  Depending  on  the  test  conditions,  copious  sources 
of  noise  might  be  present  and  must  be  considered  in  the  test  design.  Also, 
test  resources  and  programmatic  constraints  may  prohibit  common 
designs.  Consider  the  basic  principles  of  DOE  when  designing  a  test:  ran¬ 
domization,  replication,  and  blocking.  Randomization  is  the  underlying 
foundation  of  the  use  of  statistical  methods.  It  reduces  the  likelihood  of 
introducing  bias  to  the  experiment  by  randomizing  the  effect  of  uncon¬ 
trolled  variables,  such  as  unplanned  weather  effects.  Replication  of  test 
points  allows  for  estimation  of  system  variability  and  test  procedure 
error.  Blocking  provides  another  way  to  address  variability  and  improves 
the  power  to  detect  a  factor  effect  (Freeman  et  al.,  2011).  Coleman  and 
Montgomery  (1993)  provide  guidelines  in  the  preexperimental  planning 
phases  to  assist  with  designing  and  conducting  an  experiment.  Some 
other  papers  useful  in  explaining  experimental  planning  and  design 
include  Hunter  (1977)  and  Montgomery  (2005). 
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Phase  2:  Analyze  and  Build 

If  all  steps  leadingto  this  phase  are  properly  and  thoroughly  planned, 
then  the  test  is  well  defined.  However,  no  test  execution  goes  as  planned 
due  to  nuisance  factors  and  noise  that  exist  such  as  weather  and/or  data 
processing.  Data  are  collected  at  this  phase  and  analyzed  by  the  test  team. 
A  mathematical  model  is  created  for  each  response  variable  by  mapping 
a  response  surface  over  the  region  of  interest  (operational  range)  so  that 
the  effect  of  factors  on  that  response  can  be  studied  (Johnson  et  al.,  2012). 
The  analysis  will  result  in  generating  statistically  defensible  models  that 
inform  the  decision  maker. 

Phase  3:  Report 

The  test  team  should  draw  conclusions  based  on  information 
extracted  from  test  data.  Appropriate  scientific  test  and  analysis  tech¬ 
niques  should  be  employed  so  that  senior  leaders  can  make  an  informed 
decision  backed  by  defensible  data. 

Brief  Review  of  Two-Sided  Tolerance  Intervals 

In  this  section,  two-sided  approximation  tolerance  intervals  are  con¬ 
sidered.  To  describe  a  general  two-sided  tolerance  interval  form,  let  xp 
x„, ..,  x  be  values  of  a  random  sample  X,,X„, ..,  X  of  size  n  from  a  normal 
distribution  with  mean  p  and  variance  o2  where: 

X  ~  N(p,  a2) 

The  100(P)%  two-sided  tolerance  interval  with  confidence  100(l-a)% 
is  of  the  form  x  ±  k2s  for  which  the  following  applies: 

Pr[P(x  -  k2s  <  X  <  x  +  k2s)>P ]  =  1  -  a 

where  x  is  the  sample  mean,  k2  is  a  constant  multiplier,  s  is  the  sam¬ 
ple  standard  deviation,  1-a  is  the  confidence  level  associated  with  the 
interval,  and  P  is  the  proportion  of  distribution  covered  by  the  interval, 
referred  to  as  coverage.  To  describe  the  two-sided  regression  tolerance 
interval,  let’s  consider  the  general  structure  for  a  regression  model: 

VrP  0+PXi+  M  = 
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where  y.  is  the  pxl  response  vector,  x.  is  the  known  mxl  factor  variable 
vector,  /?  is  the  pxl  intercept  vector,  /?  is  unknown  p  x  m  regression 
parameter  vector,  and  8.  is  assumed  to  be  a  vector  of  independent,  nor¬ 
mally  distributed  error  terms,  each  with  mean  zero  and  variance  a2.  To 
estimate  //,  least  squares  regression  is  applied  based  on  a  set  of  n  observa¬ 
tions.  The  predicted  mean  response  would  then  be  of  the  form: 


y  =  xp, 


Suppose  for  any  known  value  of  factors  x  =  x.  with  a  corresponding 
fitted  value  y. ,  the  100(P)%  two-sided  tolerance  interval  with  confidence 
100(l-a)%  is  of  the  form: 


where  k  is  the  tolerance  factor  and  s2  is  the  residual  mean  square  error 
based  on  degrees  of  freedom. 

Monte  Carlo  Evaluation  of  Tolerance  Intervals 

The  first  approximation  considered  was  proposed  by  Howe  (1969). 
Howe  introduced  an  approximate  factor  for  a  two-sided  tolerance  inter¬ 
val  for  a  normally  distributed  population  given  as: 


where  is  xj.a  df  the  ath  percentile  of  the  chi-square  distribution  with 
df  degrees  of  freedom,  n  is  the  sample  size,  df  =  n  -  m  (number  of  inde¬ 
pendent  random  samples)  is  degrees  of  freedom  defined  as  the  number  of 
values  that  are  free  to  vary,  and  z(i-p)/2  is  the  pth  percentile  of  the  standard 
normal  distribution. 

The  second  approximation  was  proposed  by  Zorn,  Gibbons,  and 
Sonzogni  (1997).  They  introduced  a  weighted  tolerance  interval  for 
estimating  detection  and  quantification  limits  in  the  chemical  field. 
Leveraging  the  earlier  work  of  Lieberman  and  Miller  (1963)  in  developing 
simultaneous  tolerance  intervals  for  linear  regression,  they  translated 
it  to  a  nonsimultaneous  case.  The  two-sided  approximation  (TI2)  would 
result  in: 
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where  xQ  is  a  point  in  the  design  space,  X  is  the  design  matrix  of  the 
regression  model,  <£> 1  ( P )  is  the  inverse  cumulative  normal  distribution, 
and  £f,d/is  the  Student’s  ^-inverse  cumulative  distribution  function  using 
degrees  of  freedom  for  the  corresponding  confidence. 


The  third  approximation  is  credited  to  De  Gryze  et  al.  (2007)  when 
they  proposed  taking  a  in  both  y2  ( df)  and  t(df)  quantiles,  thus  resulting 
in  the  approximation  below: 


Vn±s 


la,df 


JxT(XrXH+0-1(P) 


/  df 
IXHl-a-df) 


where  xQ  is  a  point  in  the  design  space,  X  is  the  design  matrix  of  the 
regression  model,  @ 4  (P)  is  the  inverse  cumulative  normal  distribution, 
and  t^,df)  is  the  Student’s  t-inverse  cumulative  distribution  function 
using  degrees  of  freedom  for  the  corresponding  confidence. 


The  final  method  introduced  is  the  exact  two-sided  tolerance  interval 
due  to  Krishnamoorthy  and  Mathew  (2009).  The  k  is  the  solution  of  the 
integral  equation: 

2m J0"  P  WV))  j  (2 <D(z)  -  ir-V/;(2)  =  1  -  a 

where  df  is  the  degrees  of  freedom,  d2=  x'(XX)1  x  where  X  is  the  design 
matrix  of  the  linear  regression  model,  m  is  the  number  of  independent 
random  samples  (factors),  <1>(2)  is  the  cumulative  distribution  function, 
<f>(z )  is  the  probability  density  function,  Fyf  is  the  cumulative  distribu¬ 
tion  function  of  a  chi-square  distribution  with  df  degrees  of  freedom. 
Detailed  derivation  of  the  exact  equation  can  be  found  in  Howe  (1969), 
equations  1.2.3, 1.2.4, 2.5.7,  and  2.5.8)  and  Witkovsky  (2013).  This  article 
employs  the  MATL  AB  tolerance  package  developed  by  Witkovsky  (2009) 
using  the  Gauss-Kronrod  quadratic  formulae  for  integration. 

The  following  Monte-Carlo  simulation  algorithm  is  applied  to 
approximations  by  De  Gryze  et  al.  (2007),  Howe  (1969),  and  Zorn  et  al. 
(1997). 


1.  Simulate  500  design  points  (ar0)  within  the  test  space  uniformly 
distributed  throughout  the  design  space. 
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2.  For  a  given  1-a,  p,  compute  tolerance  interval  multiplier  at  xQ 
(design  point  in  the  test  space). 

3.  Since  the  tolerance  interval  multiplier  is  a  function  of  the  posi¬ 
tion  in  the  design  space,  take  the  average  tolerance  interval 
multiplier  value  (based  on  the  Monte  Carlo  simulation  sample  of 
500  points  previously  mentioned)  and  multiply  by  tll  dfto  deter¬ 
mine  the  tolerance  factor. 


Next,  the  following  are  computed  and  compared:  (a)  the  approximate 
factor  by  De  Gryze  et  al.  (2007),  (b)  the  approximate  factor  by  Zorn  et 
al.  (1997),  (c)  the  approximate  factor  by  Howe  (1969),  and  (d)  the  exact 
tolerance  factor.  Figure  2  depicts  the  comparison  case  of  p  =  0.99  and  1-a 
=  {0.90.  0.95,  0.99}. 


FIGURE  2.  P  =  0.99,  1-a  =  (0.90,  0.95,  0.99} 
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Figure  2  shows  that  all  tolerance  factors  decrease  as  degrees  of 
freedom  increase.  Immediately  apparent  is  that  the  Zorn  et  al.  (1997) 
approximation  is  the  most  conservative,  especially  for  smaller  degrees  of 
freedom.  All  numerical  results  are  slightly  above  the  exact  method,  but 
the  Howe  (1969)  approximation  is  slightly  lower  with  larger  degrees  of 
freedom.  The  two-sided  tolerance  intervals  performed  well  for  degrees 
of  freedom  4  and  above.  The  approximated  values  performed  well  against 
the  exact  method,  but  it  can  be  noted  that  Zorn’s  method  would  require 
a  larger  sample  size.  In  general,  to  cover  a  multifactor  region  requires  a 
wider  tolerance  region  compared  to  the  normal  sample.  So  why  inves¬ 
tigate  approximation  methods  when  an  exact  method  is  available?  An 
exact  method  calculation  can  be  extremely  complex  and  is  rarely  if  ever 
available  on  statistical  software  packages.  It  can  also  be  thought  of  as 
costly,  given  the  difficulty  and  time  involved  in  obtaining  an  exact  solu¬ 
tion.  Therefore,  approximation  methods  are  generally  preferred,  but  their 
accuracy  is  seldom  confirmed.  If  an  approximation  needs  to  be  used, 
the  author  recommends  the  De  Gryze  et  al.  (2007)  proposed  approxi¬ 
mate  method  as  a  statistical  test  analysis  method  commensurate  with 
designed  experiments.  The  appeal  of  the  De  Gryze  et  al.  (2007)  method 
stems  from  the  fact  that  this  method  takes  into  account  the  design 
structure  and  variance,  is  easier  to  compute,  and  is  comparable  to  the 
exact  method. 

Use  of  Tolerance  Intervals  in  Designed 
Experiment  Case  Study 

This  section  will  apply  the  two-sided  tolerance  interval  to  a  designed 
experiment  using  a  notional  case  study.  The  case  study  used  throughout 
the  article  is  for  academic  purposes  and  is  by  no  means  representative 
of  any  existing  weapon  employed  by  the  DoD.  Some  aspects  of  the  case 
study  have  been  simplified  for  educational  purposes. 

Phase  1:  Plan  and  Design 

Objective.  The  objective  of  the  experiment  is  to  characterize  the 
performance  of  a  new  and  old  air-to-ground  missile. 

Response  variables,  factors,  and  levels.  Figure  3  and  Table  1 
show  the  factors  and  levels  generated  by  the  test  team  during  the  test 
design  planning  phase.  The  response  variables  are  miss  distance  and 
impact  velocity  error. 
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FIGURE  3.  FACTORS  AND  LEVELS  GENERATED  BY  TEST  TEAM 
DURING  TEST  DESIGN  PLANNING  PHASE 
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TABLE  1.  FACTORS  AND  LEVELS 


Factor 

Levels 

A 

Variant 

0  (Legacy),  1  (New) 

B 

Range 

-1. 1 

C 

Altitude 

25,  35 

D 

Airspeed 

0.85,  0.95 

Test  design.  A  24  factorial  test  design,  16  runs  were  selected  for  this 
case  study. 

Phase  2:  Analyze  and  Build 

Test  execution.  Suppose  the  test  team  executed  the  test,  and 
Table  2  reflects  the  results  collected. 
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Model  building.  A  regression  model  was  built  and  analyzed.  The 
overall  response  models  were  significant;  range  and  airspeed  were  the 
two  most  important  factors  in  characterizing  the  air-to-ground  missile 
performance,  and  there  was  no  statistical  difference  between  the  legacy 
and  new  variant  across  the  operational  envelope.  However,  for  this 
research,  analysis  will  be  limited  to  the  tolerance  interval  computation 
under  high-airspeed  and  high-range  conditions. 


TABLE  2.  AIR-TO-GROUND  CASE  STUDY  TEST  DESIGN  AND 
RESULTS 


Run 

A: 

Variant 

B: 

Range 

C: 

Altitude 

D: 

Airspeed 

Miss 

Distance 

Impact 

Velocity 

Error 

1 

1 

-1 

25 

0.95 

3.44 

1.76 

2 

0 

1 

25 

0.95 

20.09 

18.96 

3 

0 

1 

35 

0.85 

5.63 

3.4 

4 

1 

-1 

35 

0.95 

8.58 

6.71 

5 

1 

-1 

35 

0.85 

1.14 

0.76 

6 

1 

1 

35 

0.95 

20.81 

18.46 

7 

0 

-1 

25 

0.85 

4.65 

2.83 

8 

1 

1 

25 

0.85 

4.45 

2.49 

9 

1 

1 

25 

0.95 

19.9 

17.51 

10 

1 

1 

35 

0.85 

5.44 

3.86 

11 

0 

1 

35 

0.95 

22.47 

20.35 

12 

0 

-1 

35 

0.85 

3.55 

1.61 

13 

1 

-1 

25 

0.85 

3.04 

1.38 

14 

0 

-1 

35 

0.95 

13.76 

11.45 

15 

0 

-1 

25 

0.95 

7.58 

5.39 

16 

0 

1 

25 

0.85 

5.23 

3.7 

Phase  3:  Report 

Based  on  the  models  generated,  the  following  values  were  obtained: 
mean  miss  distance  response  value  =  9.36  feet,  miss  distance  standard 
deviation  =  2.33  feet,  mean  impact  velocity  error  response  value  =  7.54 
feet/s,  impact  velocity  error  standard  deviation  =  2.15  feet/s,  and  degrees 
of  freedom  =  15.  Suppose  the  test  team  has  selected  a,  confidence  =  0.05 
(95%)  and  p,  proportion  of  population  =  99%.  Referring  to  the  De  Gryze 
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et  al.  (2007)  approximation,  the  test  team  was  able  to  report  “with  95% 
confidence,  at  least  99%  of  the  miss  distance  population  will  be  between 
5.0  to  26.3  ft.,  and  at  least  99%  of  the  velocity  error  population  will  be 
between  3.9  ft./s  to  23.5  ft./s  under  the  specified  condition.”  Now  the 
test  team  overlaid  the  two  most  important  factors— range  and  airspeed— 
and  bounded  the  “test  design  space”  with  tolerance  intervals  obtained 
previously  for  condition  1  (Figure  4).  From  this  plot,  the  team  can  easily 
extract  the  “sweet  spot,”  “operating  window,”  or  in  this  case  the  “toler¬ 
ance  interval  space”  where  they  can  ascertain  with  a  specified  confidence 
that  at  least  99  percent  of  both  responses  would  be  found,  under  the 
specified  conditions. 

FIGURE  4.  RANGE  AND  AIRSPEED  OVERLAY  PLOT 


Further,  the  test  team  investigated  how  confidence  and  tolerance 
intervals  compared.  The  95  percent  confidence  interval  for  miss  distance 
mean  and  impact  velocity  error  mean  were  found  to  be  within  13.8  to  17.5 
feet,  and  12  to  15.4  feet/s,  respectively.  This  means  the  “true”  mean  of 
the  miss  distance  and  impact  velocity  error  measurements  lies  within 
these  bounds.  Oftentimes,  we  might  not  need  to  place  bounds  on  the 
distribution  parameters,  but  on  the  specified  proportion  of  population 
instead,  hence  the  appeal  of  tolerance  intervals.  The  confidence  interval 
may  win  the  interval  popular  vote;  however,  the  beauty  of  the  tolerance 
interval  lies  in  the  fact  it  takes  into  account  not  only  the  sample  size,  but 
also  the  estimates  of  mean  and  standard  deviation  noise.  Given  the  test 
data  generated,  the  test  team  was  able  to  narrow  down  and  recommend 
a  specific  response  interval  where  99  percent  of  the  population  would  lie 
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at  the  factors  identified  under  the  specified  conditions.  This  enabled  the 
program  manager  to  set  an  operational  window  where  the  air-to-ground 
missile  would  perform  at  its  optimum  for  high  airspeeds  and  high  range. 
(Recall  the  case  study  is  notional,  but  this  is  an  illustration  of  the  type  of 
information  that  can  be  drawn.) 

In  this  case  study,  a  statistical  tolerance  interval  ensured  a  defensible 
conclusion  with  a  sound  analytical  basis,  rather  than  simply  stating  the 
mean  as  criticized  in  the  DOT&E  FY 2012  Annual  Report.  Through  the 
combined  use  of  DOE,  regression  analysis,  and  tolerance  intervals,  T&E 
practitioners  are  able  to  frame  the  operating  window  with  some  confi¬ 
dence  and  have  the  ability  to  map  out  the  test  space  where  factors  can 
be  reliably  tested.  This  is  a  significant  improvement  over  simply  stating 
a  single  average  across  all  test  conditions,  and  it  allows  us  to  extract 
more  information  from  limited  resources  and  test  events.  The  efficien¬ 
cies  obtained  through  the  meticulous  planning  using  DOE  principles 
were  retained.  An  advanced  statistical  analysis  that  complements  DOE 
proved  capable  of  defining  an  operating  window  with  some  certainty  and 
well-understood  risks  where  the  air-to-ground  missile  can  be  adequately 
operated.  Understanding  the  appropriate  use  of  statistical  analysis 
technique  is  imperative  and  does  matter;  for  example,  interaction  effects 
need  to  be  considered  and  a  simple  one-way  analysis  of  variance  or  use  of 
average  value  might  ignore  or  hide  the  interaction  between  main  effects. 
Therefore,  the  research  into  suitable  advanced  statistical  analysis  meth¬ 
ods  commensurate  with  DOE  needs  to  continue. 


Through  the  combined  use  of  DOE,  regression 
analysis,  and  tolerance  intervals,  T&E 
practitioners  are  able  to  frame  the  operating 
window  with  some  confidence,  and  have  the  ability 
to  map  out  the  test  space  where  factors  can  be 
reliably  tested. 
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Limitations  and  Future  Research 

In  general,  tolerance  intervals  offer  a  better  means  to  assure,  with 
some  confidence,  that  a  fixed  proportion  of  the  systems’  performance  over 
the  design  space  falls  within  a  specified  interval.  The  analysis  method 
reaps  the  benefits  of  a  designed  experiment  and  employs  statistical  tech¬ 
niques  that  are  commensurate  with  DOE.  This  research,  however,  does 
have  limitations,  indicating  a  need  for  further  discussion  and  research. 
Future  research  should  include  qualitative  metrics,  such  as  categorical 
factors.  In  addition,  other  tolerance  intervals  such  as  nonparametric 
regression  tolerance  intervals  should  be  investigated  for  future  use 
in  the  defense  test  community  (see  Young,  2010,  for  other  intervals). 
One-sided  regression  tolerance  intervals  for  defense  testing  should  be 
presented  and  compared  using  the  proposed  Monte  Carlo  simulation 
algorithm;  the  calculation  is  generally  simpler  than  the  two-sided  case. 
When  exact  methods  are  not  available,  the  author  recommends  using 
the  approximate  methods  mentioned  in  this  article  that  are  best  suited 
for  multiple  regression  models.  Be  forewarned  that  the  use  of  tolerance 
intervals  may  require  a  larger  sample  size;  for  this  reason  and  to  properly 
size  your  experiment,  the  author  also  recommends  investigating  the  use 
of  tolerance  intervals  in  test  planning  (see  Whitcomb  &  Anderson,  2011, 
for  examples). 

Recommendations  and  Conclusions 

The  defense  T&E  community  has  progressed  in  its  efforts  to  advance 
statistical  rigor  within  the  community  over  the  past  3  years;  however, 
some  areas  still  need  improvement.  One  area  is  to  improve  interaction 
between  necessary  stakeholder  organizations  and  the  T&E  community. 
All  organizations  that  have  an  impact  and/or  influence  on  the  program’s 
T&E  planning,  execution,  and  assessment  need  to  be  engaged  in  the  T&E 
WIPT  as  early  as  possible.  Another  area  would  be  to  increase  education 
and  training  on  the  use  of  STAT  for  all  stakeholders,  and  this  means  going 
above  and  beyond  what  confidence  intervals  provide.  Finally,  best  prac¬ 
tices,  lessons  learned,  and  research  need  to  be  continuously  published 
and  readily  available  to  the  T&E  community. 

In  these  resource-constrained  times,  every  dollar  spent  on  defense 
must  count.  As  the  DoD  moves  toward  generating  defensible  data  through 
the  use  of  DOE  for  test  designs  and  institutionalizing  statistical  rigor 
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within  the  T&E  community,  it  seems  logical  to  employ  advanced  statisti¬ 
cal  analysis  methods  that  reap  the  benefits  afforded  by  DOE  to  generate 
efficiencies.  Rigor  should  not  end  with  the  test  design,  and  solid  analyti¬ 
cal  evidence  needs  to  be  presented  all  the  way  through  test  reporting. 
The  literature  to  date  does  not  adequately  address  the  appropriate  use  of 
defensible  data  developed  through  improved  design  methods,  nor  does  it 
propose  a  statistical  analysis,  such  as  tolerance  intervals,  commensurate 
with  test  designs  developed  using  DOE  for  the  defense  community.  This 
article  fills  that  gap  by  introducing  the  applicability  of  tolerance  intervals 
as  an  analysis  technique  in  a  designed  experiment  and  by  comparing 
several  two-sided  approximate  tolerance  factors  estimated  by  Monte 
Carlo  simulation  to  the  exact  method.  Further,  this  article  provides  a 
recommendation  of  the  most  appropriate  tolerance  interval  and  its  appli¬ 
cability  to  the  defense  T&E  community  using  a  simple  case  study.  This 
analytical  method  provides  a  meaningful  objective  way  to  add  rigor  to 
an  otherwise  subjective  assessment,  extracts  more  information  to  state 
how  the  system  will  perform  in  the  operational  conditions,  and  serves  as 
a  quantitative  decision  aid  to  our  senior  leaders. 
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The  U.S.  Department  of  Defense  endorsed  and  later 
mandated  the  use  of  Technology  Readiness  Assess¬ 
ments  (TRAs)  and  knowledge-based  practices  in  the 
early  2000s  for  use  as  a  tool  in  the  management  of 
program  acquisition  risk.  Unfortunately,  implementing 
TRAs  can  be  costly,  especially  when  programs  include 
knowledge-based  practices  such  as  prototyping,  perfor¬ 
mance  specifications,  test  plans,  and  technology  maturity 
plans.  What  is  the  economic  impact  of  these  TRA  prac¬ 
tices  on  the  past  and  present  acquisition  performance 
of  the  U.S.  Army,  Navy,  and  Air  Force?  The  conundrum 
today  is  that  no  commonly  accepted  approach  is  in  use 
to  determine  the  economic  value  of  TRAs.  This  article 
provides  a  model  for  the  valuation  of  TRAs  in  assessing 
the  risk  of  technical  maturity. 
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Background 

The  U.S.  Government  Accountability  Office  (GAO),  formerly  the 
General  Accounting  Office,  has  reported  on  the  acquisition  performance 
of  major  defense  acquisition  programs  (MDAPs)  since  1960  (U.  S.  General 
Accounting  Office,  1988).  From  the  inception  of  the  GAO’s  mandate  to 
report  annually  to  Congress  on  its  assessment  findings,  the  ability  of  the 
U.S.  Department  of  Defense  (DoD)  to  consistently  execute  its  acquisition 
plan  for  the  purchase  of  major  weapon  systems  has  been  erratic,  seldom 
meeting  cost,  schedule,  or  original  performance  objectives.  From  1997 
to  2012,  the  DoD's  budget  grew  by  almost  200  percent  to  $529  billion, 
representing  more  than  20  percent  of  the  total  operating  budget  of  the 
U.S.  government  (DoD,  2013b).  Amazingly,  31  percent  of  all  MDAPs  since 
1997  have  incurred  either  a  significant  or  critical  Nunn-McCurdy  cost 
breach  (DoD,  2013c).  In  addition,  during  1995-2013  each  of  the  military 
services  has  experienced  cancellation  of  several  major  programs  without 
receiving  any  or  very  few  operational  units  for  the  funds  expended  (DoD, 
2013c).  Specifically,  the  Army  cancelled  14  MDAPs  (Table  1): 


TABLE  1.  ARMY-CANCELLED  MDAPs  (1995-2013) 


1 

Aerial  Common  Sensor  (ACS) 

8 

Joint  Common  Missile  (JCM) 

2 

Armed  Reconnaissance 

Helicopter  (ARH) 

9 

Joint  Tactical  Radio  System- 
Ground  Mobile  Radio  (Army 
Portion)  (JTRS-GMR) 

3 

Army  Tactical  Missile 
System-Brilliant  Anti-armor 
Technology  (ATACMS-BAT) 

10 

Land  Warrior  Integrated  Soldier 
System 

4 

C-27J  Military  Transport 
Aircraft  (Army  Portion) 

11 

Net-Enabled  Command 

Capability  (NECC) 

5 

RAH-66  Comanche 

Reconnaissance  Armed 

Helicopter 

12 

Non  Line-of-Sight-Land 

Systems  (NLOS-LS) 

6 

XM2001  Crusader  Self- 

Propelled  Howitzer 

13 

Patriot  Medium  Extended  Air 

Defense  System  Combined 
Aggregate  Program  Fire  Unit 
(Patriot  MEADS  CAP  Fire  Unit) 

7 

Future  Combat  System  (FCS) 

14 

Surface  Launched  Advanced 

Medium  Range  Air-to-Air 

Missile  (SLAMRAAM) 
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The  Navy  cancelled  seven  MDAPs  (Table  2): 


TABLE  2.  NAVY-CANCELLED  MDAPs  (1995-2013) 


1 

Advanced  Deployable  System 
(ADS) 

5 

Extended  Range  Munition 
(ERM) 

2 

Advanced  SEAL  Delivery 
System  (ASDS) 

6 

F-35  Alt  Engine  (Navy  Portion) 

3 

Expeditionary  Fighting  Vehicle 
(EFV) 

7 

VH-71  Kestrel  Presidential 

Helicopter 

4 

Electronic  Patrol  -  X  (EP-X) 

Finally,  the  Air  Force  cancelled  10  MDAPs  (Table  3): 


TABLE  3.  AIR  FORCE-CANCELLED  MDAPs  (1995-2013) 


1 

Third  Generation  Infrared 

System  (3GIRS) 

6 

Expeditionary  Combat  Support 
System  (ECSS) 

2 

C-130  Avionics  Modernization 

Program  (AMP) 

7 

F-35  F136  Engine 

3 

C-27J  Joint  Cargo  Aircraft 

8 

National  Polar-Orbiting 
Operational  Environmental 
Satellite  System  (NPOESS) 

4 

Combat  Search  and  Rescue 

(CSAR-X) 

9 

Space  Based  Space 

Surveillance  (SBSS)  Follow-on 

5 

E-10  Multi-Sensor  Command 

and  Control  Aircraft  (E-10 
MC2A) 

10 

Transformational  Satellite 

Communications  System 
(TSAT) 

In  1999  the  U.S.  General  Accounting  Office  defined  a  framework 
of  acquisition  practices  modeled  after  commercial  best  practices  that 
emphasized  knowledge-based  decision  making,  and  recommended  its 
adoption  by  the  DoD  (U.S.  General  Accounting  Office,  1999).  The  DoD 
adopted  knowledge-based  practices  in  2001  with  the  issuance  of  DoD 
Directive  (DoDD)  5000.1  and  DoD  Instruction  (DoDI)  5000.2  (see  DoD 
2013e;  2013a,  respectively).  Starting  in  May  2003,  and  annually  thereaf¬ 
ter,  the  GAO  has  reported  to  Congress  its  assessment  of  the  acquisition 
performance  of  MDAPs,  emphasizing  the  DoD’s  use  of  mature  tech¬ 
nologies  Technology  Readiness  Assessments  (TRAs).  Such  assessment 
includes  adherence  to  knowledge-based  acquisition  practices  such  as 
prototyping,  performance  specifications,  test  plans,  and  technology 


Defense  ARJ,  October  2014,  Vol.  21  No.  4: 826-850 


828 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


maturity  plans  (U.  S.  General  Accountability  Office,  2005,  2006,  2007, 
2008a,  2009,  2010,  2011,  2012;  U.  S.  General  Accounting  Office,  2003, 
2004).  The  DoD  attributes  a  significant  proportion  of  poor  acquisition 
performance  to  the  incorporation  of  immature  technologies  into  its 
weapon  system  acquisitions  by  DoD  components,  Defense  agencies,  and 
their  suppliers.  Associated  with  the  DoD's  yearly  multibillion  dollar  bud¬ 
get  for  the  procurement  of  military  weapon  systems  is  the  expenditure  of 
millions  of  dollars  each  year  performing  TRAs  as  one  of  the  approaches 
to  monitor  and  control  the  perceived  risk  of  incorporating  immature 
technology  into  the  acquisition  process.  The  DoD  uses  TRAs  as  a  means 
of  identifying  key  components,  referred  to  as  critical  technologies  (CT), 
and  assessing  their  maturity  using  a  nine-point  Technology  Readiness 
Level  (TRL)  scale  (Mankins,  1995).  As  part  of  a  TRA,  an  independent 
team  of  subject  matter  experts  assists  the  program  manager  (PM)  in 
the  process  of  identifying  CTs  believed  to  be  the  major  drivers  of  cost 
and  schedule  performance  during  the  acquisition.  The  team  also  assists 
the  PM  in  assessing  component  maturity  and  assigning  TRLs.  Their 
assessment  is  then  documented  in  a  TRA  report  prior  to  the  major  deci¬ 
sion-making  juncture  in  the  overall  acquisition  life  cycle  (i.e.,  Milestone 
B).  (Note  that  the  TRA  report  is  mandated  by  the  Milestone  Decision 
Authority  [DoD,  2011a,  2011b].)  Typically  CTs  are  advanced  or  leading- 
edge  technology  that  will  push  the  performance  envelope  of  the  weapon 
system,  thus  providing  a  strategic  military  advantage  (Petraeus,  2010). 
The  DoD  believes  that  identifying  and  mitigating  the  use  of  immature 
technologies  (i.e.,  TRL  <  6)  early  is  the  key  to  improving  overall  acquisi¬ 
tion  performance  (i.e.,  reducing  cost  and  schedule  overruns,  increasing 
delivery  order  quantities,  successful  weapon  systems  deployment,  etc.) 
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(Cancian,  2010;  DoD,  2009;  U.  S.  General  Accounting  Office,  1998, 1999). 
Bailey,  Mazzuchi,  Sarkani,  and  Rico  (2014)  reported,  however,  that  dur¬ 
ing  2003-2012  only  slightly  more  than  half— 58.1  percent— of  the  CTs 
being  used  in  development  acquisitions  were  sufficiently  matured  (i.e., 
TRL  >  6)  (see  Table  4).  This  tendency  to  proceed  into  development  or 
production  with  less  knowledge  than  required  has  led  to  similar  results 
experienced  over  the  last  five  decades,  with  several  programs  failing  to 
meet  the  original  cost,  schedule,  and  performance  objectives  (Bair,  1994; 
Fox,  2011;  U.  S.  General  Accounting  Office,  1988). 


TABLE  4.  2003-2012  DoD  CT  MATURITY  ASSESSMENTS 


Year 

Critical  Technologies 

Immature  Total 

Mature 

2012 

103 

345 

70.1% 

2011 

106 

371 

71.4% 

2010 

105 

372 

71.8% 

2009 

177 

420 

57.9% 

2008 

208 

466 

55.4% 

2007 

241 

451 

46.6% 

2006 

225 

428 

47.4% 

2005 

251 

443 

43.3% 

2004 

193 

391 

50.6% 

2003 

39 

117 

66.7% 

Avg 

165 

380 

58.1% 

Sources:  U.S.  General  Accounting  Office,  2003,  2004;  GAO  2005,  2006,  2007,  2008a, 
2009,  2010,  2011,  2012 


Utilization  of  proven  technologies  that  offer  moderate  performance 
improvements,  yet  are  well  understood  in  terms  of  meeting  scope,  cost, 
schedule,  and  performance  constraints,  is  DoD's  preferred  acquisition 
approach.  Currently,  however,  basic  arguments  favor  applying  the  five- 
stage  DoD  acquisition  life  cycle  defined  by  DoD  (2013d).  This  life  cycle 
includes  up-front  investments  in  large-scale  system  prototypes  during 
the  Technology  Demonstration  (TD)  phase;  and  the  performance  of 
TRAs,  along  with  identifying  their  associated  CTs,  assigning  TRLs, 
and  ensuring  they  reach  sufficient  maturity— all  are  qualitative  at  best 
and  are  based  only  on  engineering  judgment  or  face  validity.  Clausing 
and  Holmes  (2010)  devised  a  structured  technology  readiness  method 
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that  added  quantification  measures  in  an  attempt  to  remove  perceived 
subjectivity  within  the  National  Aeronautics  and  Space  Administration's 
TRL  framework.  However,  little  quantitative  evidence  has  been  collected 
on  the  actual  economic  benefits  of  technology  maturity  via  TRAs  for 
any  of  the  military  Services.  Therefore,  the  purpose  of  this  article  is  to 
examine  the  wealth  of  information  emerging  from  government  agencies 
such  as  the  GAO,  DoD,  and  others  and  apply  economic  models  to  begin 
examining  the  quantitative  benefits  of  technology  maturity  for  the  major 
programs  of  each  of  the  military  Services.  The  results  of  this  analysis 
should  help  members  of  the  acquisition  community  determine  whether 
TRA  knowledge-based  practices  have  a  positive  effect  on  acquisition  out¬ 
comes.  More  important,  in  today's  environment  of  fiscally  austere  federal 
budgets  and  after  the  impact  of  the  sequester,  such  evidence  may  also  be 
of  benefit  to  military  strategists  if  the  use  of  TRAs  helps  reduce  cost  and 
schedule  overruns  and  increases  delivery  order  quantities  (DoQs)  for  the 
Army,  Navy,  and  Air  Force  (Petraeus,  2010). 


Utilization  of  proven  technologies  that  offer 
moderate  performance  improvements,  yet  are 
well  understood  in  terms  of  meeting  scope,  cost, 
schedule,  and  performance  constraints,  isDoD's 
preferred  acquisition  approach. 


Problem 

The  DoD  portfolio  of  MDAPs  currently  stands  at  95,  for  fiscal  year 
2013,  with  an  estimated  cost  for  development  and  procurement  of  nearly 
$1.7  trillion  (DoD,  2013b).  Overall  acquisition  performance  has  been  less 
than  stellar,  and  a  significant  proportion  of  the  programs  suffer  from 
excessive  cost,  schedule  overruns,  and  dramatically  reduced  DoQs, 
while  some  have  been  cancelled  outright.  The  DoD's  position  is  that 
technology  maturity,  or  lack  thereof,  is  a  primary  measure  of  acquisition 
performance  (GAO,  2008b).  That  is,  weapon  systems  that  use  mature 
technologies  will  have  better  acquisition  performance  than  those  using 
immature  technologies  (Weapon  Systems  Acquisition  Reform  Act, 
2009).  Technical  maturity  (or  knowledge-based  practices  as  they  are 
frequently  called)  such  as  the  TD  phase,  full-scale  system  prototype,  and 
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the  TRA  process  together  may  cost  up  to  10  percent  of  the  acquisition 
budget  through  the  manufacturing  phase.  The  fundamental  concept  is 
that  these  up-front  technology  maturity  investments  will  head  off  down¬ 
stream  manufacturing,  operating,  and  maintenance  costs  (Assessment 
Panel,  2006;  DoD,  2008;  Olagbemiro,  Mun,  &  Shing,  2011).  Each  of  the 
military  Services  has  incurred  costs  implementing  the  TRA  process  as 
mandated  by  the  DoD  and  have  assumed  the  advertised  benefits  would 
lead  to  a  successful  acquisition.  However,  little  data  are  available  on 
the  economic  benefits  of  performing  TRAs.  Even  the  most  avid  support¬ 
ers  of  TRAs  want  to  quantify  their  economic  benefits  (Dubos,  Saleh,  & 
Braun,  2008;  Kenley  &  El-Khoury,  2012).  The  use  of  economic  valuation 
is  experiencing  a  revival  of  sorts  throughout  the  project  management, 
engineering,  information  technology,  and  acquisition  communities 
(Honour,  2004;  Reinertsen,  2009).  Among  these,  the  most  commonly 
cited  measure  of  business  value  is  the  concept  of  return  on  investment 
or  ROI  (Morgan,  2005).  That  is,  cumulative  economic  benefits  less  costs, 
divided  by  costs.  Today,  however,  economists  promote  a  suite  of  other, 
more  valid  measures,  such  as  net  present  value  (NPV),  internal  rate 
of  return,  real  options  analysis  (ROA),  and  numerous  other  measures 
of  project  performance  (Tockey,  2004).  The  majority  of  these  methods 
are  what  is  known  as  top-down  parametric  models,  which  require  only 
a  few  basic  inputs  such  as  costs,  benefits,  interest  rate,  time  horizon,  or 
even  risk.  Costs  and  benefits  are  the  key  inputs.  Cost  data  are  being  col¬ 
lected  with  increasing  frequency,  and  soft,  nonquantifiable  benefits  are 
sometimes  collected  as  well.  It's  only  when  the  latter  are  converted  into 
economic  terms,  or  monetized,  that  the  portfolio  of  economic  equations 
and  models  may  be  applied.  In  spite  of  the  myriad  complex  economic 
methods,  three  basic  forms  seem  to  be  standing  the  test  of  time  (i.e., 
ROI,  NPV,  and  ROA).  Therefore,  the  basic  research  problem  or  question, 
given  the  DoD  mandate  requiring  TRAs  for  all  MDAPs,  is  this:  What  are 
the  economic  benefits  of  applying  TRAs  for  each  military  Service?  More 
specifically,  what  is  the  associated  cost,  benefit,  ROI,  NPV,  and  ROA  of 
TRAs?  Consequently,  the  fundamental  goal  and  objective  of  this  article  is 
to  collect  and  analyze  MDAP  acquisition  data,  apply  some  of  these  basic 
economic  models,  and  explore  the  economic  value  of  applying  TRAs  to 
acquisitions  of  each  military  Service. 
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Research  Method 

The  research  method  developed  for  this  article  involved  collecting 
measurements,  which  were  used  to  analyze  the  value  of  TRAs  for  each 
military  Service.  It  was  used  to  help  determine  the  costs  and  benefits  of 
technology  maturity  and  whether  it  translates  to  improved  acquisition 
performance.  First,  a  spreadsheet  model  was  constructed  consisting 
of  basic  attributes,  such  as  government  agency,  program  type,  program 
name,  acquisition  costs,  and  technology  maturity.  Then  other  fundamen¬ 
tal  valuation  drivers  were  added  to  derive  key  indicators  of  value,  such  as 
acquisition  risks,  TRA  costs,  and  TRA  benefits.  Using  the  basic  acquisi¬ 
tion  attributes  and  derived  data,  metrics  and  models  were  then  added  to 
help  determine  the  value  of  TRAs.  These  included  benefit/cost  ratio  (B/ 
CR),  ROI,  NPV,  breakeven  point  (BEP),  and  ROA.  The  cost-and-benefit 
spreadsheet  was  then  populated  with  acquisition  data  from  GAO  reports 
of  the  major  programs  from  each  military  Service  covering  a  10-year 
period— 2003  to  2012— for  further  analysis.  These  seven  metrics  were 
originally  outlined  by  Rico  (2007)  as  follows: 

1.  Costs  =  Total  amount  of  money  spent  on  technology  readiness 

2.  Benefits  =  Total  amount  of  money  gained  from  technology 
readiness 

3.  B/CR  =  Ratio  of  technology  readiness  benefits  to  costs 

4.  ROI  percent  =  Ratio  of  adjusted  technology  readiness  benefits  to 
costs 

5.  NPV  =  Discounted  cash  flows  of  technology  readiness 

6.  BEP  =  Point  when  benefits  exceed  costs  of  technology  readiness 

7.  ROA  =  Business  value  realized  from  strategic  delay  due  to  risk 

ROA  attempts  to  estimate  the  value  of  the  flexibility  a  PM  has  to 
change  direction  of  a  project  as  new  data  and  information  emerge  to 
help  remove  uncertainty  about  the  viability  of  a  chosen  or  desired  path 
(de  Week,  de  Neufville,  &  Chaize,  2004).  Trigeorgis  (1993)  asserted  that 
managerial  flexibility  is  a  set  of  real  options  that  may  consist  of  options 
to  defer,  abandon,  contract,  expand,  or  switch  investment.  Each  of  these 
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options  may  result  in  a  different  valuation.  The  Black-Scholes  method  for 
determining  real  option  value  was  chosen  for  this  study  as  it  provides  the 
most  accurate  valuation  available  today.  As  suggested  earlier,  parametric 
forms  of  ROA  have  emerged  making  it  possible  to  analyze  acquisition 
performance  (Black  &  Scholes,  1973;  Kodukula  &  Papudesu,  2006). 

Data  Analysis  and  Results 

A  recent  report  by  the  GAO  provided  detailed  cost  data  of  47  DoD 
programs  from  the  2012  MDAP  portfolio,  which  is  the  first  required 
input  for  determining  the  ROI  of  TRAs  (GAO,  2012).  Risk  is  the  second 
required  input  for  determining  ROI  of  TRAs.  Management  of  risk  is  a 
key  element  in  the  operational  planning  of  any  major  system  develop¬ 
ment.  Identifying  and  quantifying  the  technological  risk  of  a  program 
may  have  been  challenging  in  the  past;  however,  the  GAO  report  on  the 
2012  MDAP  portfolio  provided  the  necessary  information  for  estimating 
risks:  (a)  total  cost,  and  (b)  technology  maturity  (GAO,  2012).  Technology 
maturity  is  the  ratio  of  immature  to  total  critical  technologies  (equally 
weighted),  which  allows  us  to  determine  technology  risk  as  the  normal¬ 
ized  rank  of  technology  maturity.  Cost  risk  is  the  normalized  rank  of 
total  costs,  which  when  combined  with  technology  risk  gives  a  combined 
risk.  Finally,  we  determine  an  overall  risk  percentage  as  the  normalized 
combined  risk.  Armed  with  these  measures,  we  are  able  to  unlock  the 
benefits  of  technology  stability  and  maturity,  and  reveal  the  third  input— 
the  economic  benefits  of  performing  TRAs.  The  GAO  report  on  2012 
MDAPs  provided  three  data  points  for  determining  the  benefits  of  TRAs: 
(a)  total  costs,  (b)  technology  maturity,  and  (c)  the  average  cost  savings 
from  technology  stability  and  maturity.  Benefits  are  a  product  of  total 
costs,  risk,  and  an  average  reported  benefit  of  29.7  percent  (GAO,  2007). 
Based  on  a  sensitivity  analysis,  benefits  were  moderated  and  smoothed  by 
the  normalized  costs.  The  following  discussion  utilizes  these  measures 
in  analyzing  the  value  of  TRAs  for  each  military  Service. 

Value  Analysis  of  TRAs  for  Army,  Navy,  and  Air  Force 
MDAPs 

A  multitude  of  economic  equations-such  as  cost,  benefit,  B/CR, 
ROI  percent,  NPV,  BEP,  and  ROA-many  of  which  were  first  introduced 
during  the  industrial  revolution,  help  determine  the  business  value  of 
an  investment  such  as  cost,  benefit,  B/CR,  ROI  percent,  NPV,  BEP,  and 
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ROA  (Rico,  2007).  ROI  percent,  which  is  a  ratio  of  benefits  to  costs  less 
the  costs,  is  one  of  the  oldest  measures  used  to  estimate  business  value 
(Phillips,  1997).  Although  having  some  similarity  with  ROI  percent,  NPV 
additionally  takes  into  account  the  time-value  of  money  (e.g.,  devalua¬ 
tion  due  to  inflation)  and  is  considered  more  realistic  and  economically 
responsible.  During  the  1970s,  ROA  emerged  as  a  measurement  approach 
to  estimate  the  value  of  investments  as  a  strategy  of  delaying  investments 
due  to  risk  presence  (Black  &  Scholes,  1973;  Kodukula  &  Papudesu,  2006). 
Rico  (2007)  posits  that  ROI  percent  is  used  for  determining  near-term 
benefits,  NPV  for  mid-term  benefits,  and  ROA  for  longer  term  benefits 
in  the  presence  of  risk.  Our  study  utilizes  all  three  vantage  points  in 
analyzing  the  acquisition  data  for  the  Army,  Navy,  and  Air  Force:  (a)  ROI 
percent,  (b)  NPV,  and  (c)  ROA  (see  Tables  5-7,  respectively). 


TABLE  5.  ILLUSTRATIVE  ARMY  ROI  DATA  FROM  2012  MDAP 
PORTFOLIO 


No. 

Program 

Cost 

Benefit 

B/ 

CR 

ROI% 

NPV 

Breakeven 

ROA 

17 

Excalibur 

$178.1 

$472.5 

2.7:1 

165.4% 

$231.1 

3.9  Years 

$333.8 

28 

JHSV 

$367.4 

$851.5 

2.3:1 

131.7% 

$369.9 

5  Years 

$567.6 

23 

Gray 

Eagle 

$515.9 

$983.3 

1.9:1 

90.6% 

$335.5 

7.7  Years 

$624.8 

4 

IAMD 

$552.9 

$963 

1.7:1 

74.2% 

$280.9 

9.8  Years 

$608.2 

31 

JTRS 

AMF 

$816.1 

$1,039.9 

1.3:1 

27.4% 

$84.4 

48.4  Years 

$660.8 

32 

JTRS 

HMS 

$835.8 

$1,035.8 

1.2:1 

23.9% 

$61.1 

68.4  Years 

$658.8 

3 

AH-64D 

Block  Ilia 

$1,073.7 

$1,141.4 

1.1:1 

6.3% 

-$85.3 

-62.9  Years 

$729.7 

Note.  All  costs  and  benefits  are  in  millions  of  dollars.  IAMD  =  Integrated  Air  and  Missile 
Defense;  JHSV  =  Joint  High  Speed  Vessel;  JTRS  AMF  =  Joint  Tactical  Radio  System 
Airborne  &  Maritime/Fixed  Station;  JTRS  HMS  =  Joint  Tactical  Radio  System  Handheld, 
Manpack,  and  Small  Form  Fit. 

Source:  GAO,  2012 
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TABLE  6.  ILLUSTRATIVE  NAVY  ROI  DATA  FROM  2012  MDAP 
PORTFOLIO 


No. 

Program 

Cost 

Benefit 

B/CR 

ROI% 

NPV 

Breakeven 

ROA 

25 

IDECM 

$82.2 

$239 

2.9:1 

190.9% 

$124.8 

3.3  Years 

$175 

38 

NMT 

$188.1 

$532.2 

2.8:1 

183.0% 

$272.7 

3.4  Years 

$385.7 

45 

VTUAV 

$261.5 

$725.7 

2.8:1 

177.5% 

$366.9 

3.6  Years 

$522 

41 

SSC 

$441.3 

$1,165.3 

2.6:1 

164.1% 

$567.8 

3.9  Years 

$821.7 

37 

MUOS 

$697.8 

$1,709.2 

2.4:1 

144.9% 

$782.2 

4.5  Years 

$1,166.3 

11 

BAMS 

$1,305.2 

$2,605.4 

2.0:1 

99.6% 

$950.8 

6.9  Years 

$1,666 

16 

E  -  2  D 

AHE 

$1,774.7 

$2,920.8 

1.6:1 

64.6% 

$754.4 

11.8  Years 

$1,843.1 

Note. 

All  costs 

and  benefits  are  in  m 

il  1  ions 

of  dollars. 

BAMS  = 

Broad  Area 

Maritime 

Surveillance;  E-2D  AHE  =  Advanced  Hawkeye;  IDECM  =  Integrated  Defensive  Electronic 
Countermeasures;  MUOS  =  Mobile  User  Objective  System;  NMT  =  Navy  Multiband 
Terminal;  ROA  =  Real  Options  Analysis;  ROI  =  Return  on  Investment;  SSC  =  Ship-to- 
Shore;  VTUAV  =  Vertical  Take-Off  and  Landing  Tactical  Unmanned  Aerial  Vehicle. 
Source:  GAO,  2012 


TABLE  7.  ILLUSTRATIVE  AIR  FORCE  ROI  DATA  FROM  2012  MDAP 
PORTFOLIO 


No. 

Program 

Cost 

Benefit 

B/ 

R0I% 

NPV 

Breakeven 

ROA 

CR 

27 

JASSM- 

$373 

$1,097 

2.9:1 

194.1% 

$576.9 

3.2  Years 

$806.5 

ER 

21 

GPS  III 

$421.1 

$1,236.7 

2.9:1 

193.7% 

$649.8 

3.2  Years 

$908.8 

19 

FAB-T 

$468.8 

$1,375.3 

2.9:1 

193.3% 

$722 

3.2  Years 

$1,010.2 

12 

C-130 

$620.4 

$1,812.7 

2.9:1 

192.2% 

$949.2 

3.3  Years 

$1,329.5 

AMP 

40 

MQ-9 

$1,191.9 

$3,429.1 

2.9:1 

187.7% 

$1,777.4 

3.4  Years 

$2,500.9 

43 

SBIRS 

$1,826.7 

$5,165.1 

2.8:1 

182.8% 

$2,645.7 

3.5  Years 

$3,742.5 

High 

33 

KC-46 

$4,412.7 

$10,464.4 

2.4:1 

137.1% 

$4,648.3 

4.7  Years 

$7,041.1 

Note. 

All  costs 

and  benefits  are  in  m 

illions 

of  dollars 

.  C-130  AMP  =  C-130 

Avionics 

Modernization  Program;  FAB-T  =  Family  of  Advanced  Beyond-Line-of-Site  Terminals; 
GPS  III  =  Global  Positioning  System  III;  JASSM-ER  =  Joint  Air-to-Surface  standoff 
Missile-Extended  Range;  KC-46  =  KC-46  Pegasus  Military  Aerial  Refueling  and  Strategic 
Transport  Aircraft;  MQ-9  =  MQ-9  Reaper  Unmanned  Aerial  Vehicle;  ROA  =  Real  Options 
Analysis;  ROI  =  Return  on  Investment;  SBIRS  High  =  Space-Based  Infrared  System  High. 
Source:  GAO,  2012 
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We  examined  the  model  results  for  each  of  the  military  Service 
portfolios.  The  Army's  acquisition  performance,  as  reported  in  its  2012 
portfolio  of  MDAPs,  exhibited  a  wide  range  of  performance.  For  example, 
the  B/CR  performance  estimate  ranged  from  1.1:1  to  2.7:1,  and  the  BEP 
from  -62.9  years  to  3.9  years.  Conversely,  the  Navy’s  B/CR  ranged  from 
1.6:1  to  2.9:1,  and  the  BEP  from  11.8  years  to  3.3  years,  while  the  Air 
Force’s  B/CR  ranged  from  2.4:1  to  2.9:1,  and  the  BEP  from  4.7  years  to 
3.2  years.  This  performance  may  indicate  a  lack  of  consistent  institu¬ 
tional  adherence  to  DoDI  5000.02  by  the  Army,  although  further  detailed 
analysis  is  required.  The  Navy’s  acquisition  performance,  as  reported  in 
its  2012  portfolio  of  MDAPs,  appears  to  exhibit  more  consistency  within 
its  portfolio  of  programs  in  reaping  the  benefits  of  TRA  knowledge- 
based  practices  than  the  Army.  In  particular,  a  higher  percentage  of  its 
programs  had  an  ROI  greater  than  100  percent,  which  may  indicate  a 
more  effective  CT  selection  process  that  leverages  sufficiently  matured 
technology  for  incorporation  into  development  programs,  although  less 
than  that  accomplished  by  the  Air  Force.  The  Air  Force’s  acquisition 
performance,  as  reported  in  its  2012  portfolio  of  MDAPs,  appears  to 
exhibit  even  more  consistency  within  its  portfolio  of  programs  in  reaping 
the  benefits  of  TRA  knowledge-based  practices  than  the  Army  or  Navy. 
Of  particular  note  is  that  a  higher  percentage  of  the  Air  Force  programs 
had  a  B/CR  measurement  of  2.8:1  or  higher;  additionally,  the  Service 
reached  its  BEP  sooner  (<  3.5  years).  This  performance  may  indicate  a 
greater  efficiency  in  program  acquisition  and  operation  due  to  greater 
adherence  to  knowledge-based  practices. 

A  sampling  of  the  case  study  analysis  performed  for  each  portfolio 
is  provided  in  the  following  discussion.  Each  case  provides  additional 
insight  into  the  potential  economic  risk  associated  either  with  or  without 
sufficiently  mature  critical  technology. 

Analysis  of  AH-64D  Block  Ilia.  The  Army’s  Apache  Block  Ilia 
program  (AB3A)  is  an  upgrade  of  the  “AH-64D  Longbow  helicopters  to 
improve  performance,  situational  awareness,  lethality,  survivability,  and 
interoperability,  and  to  prevent  friendly  fire  incidents”  (GAO,  2012,  p. 
45).  The  program  acquisition  state  for  this  study  was  in  the  early  stages 
of  the  Production  and  Deployment  (P&D)  phase.  Upon  determination  of 
costs  and  benefit  of  TRAs  for  the  AH-64D  Block  Ilia,  our  model  estimates 
values  for  the  other  five  metrics  (i.e.,  B/CR,  ROI  percent,  NPV,  BEP,  and 
ROA).  Our  analysis  of  the  B/CR  metric  reflects  a  less-than-favorable 
valuation.  It  indicates,  for  every  dollar  expended,  only  a  relatively  small 
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percent  (i.e.,  10  percent)  is  returned  as  benefit.  The  ROI  percent  valuation, 
which  also  reflects  a  simple  cost-benefit  ratio,  less  the  costs,  without  con¬ 
sideration  for  the  time  value  of  money,  shows  only  a  6.3  percent  return  on 
the  program’s  investment  in  TRA  practices,  or  $.063  saved  for  every  dol¬ 
lar  invested.  The  NPV  valuation  incorporates  the  time  value  of  money  in 
the  economic  evaluation  and  provides  the  present  value  of  the  estimated 
return.  In  a  traditional  business  decision-making  scenario,  a  positive 
difference  between  NPV  and  cost  provides  justification  to  proceed  with 
the  program  or  investment.  The  NPV  valuation  here  reflects  a  significant 
negative  valuation  of -$85.3  million,  and  may  provide  sufficient  justifica¬ 
tion  to  halt  the  program  or  investment.  The  BEP  valuation  reflects  being 
unable  to  recoup  the  full  initial  cost  of  investment  in  TRA  practices 
due  to  the  remaining  immaturity  of  critical  technologies  and  unstable 
requirements.  Finally,  the  ROA  valuation  results  in  an  estimated  return 
of  $729.7  million,  which  is  $674.4  million  more  than  the  NPV  estimate, 
and  is  an  estimated  $344.7  million  less  than  the  cost  of  implementing 
TRA  practices.  In  this  example  of  the  AB3A  program,  several  of  the  key 
ROI  metrics  suggest  an  unstable  technology  base  and  cost  risk,  and  that 
the  program  risk  in  proceeding  into  the  next  phase  is  high. 

Analysis  of  Mobile  User  Objective  System  (MUOS).  The  Navy’s 
MUOS  is  “a  satellite  communications  system  that  is  expected  to  pro¬ 
vide  a  worldwide,  multi-Service  population  of  mobile  and  fixed-site 
terminal  users  with  increased  narrowband  communications  capacity 
and  improved  availability  for  small  terminal  users”  (GAO,  2012,  p.  111). 
The  program  acquisition  state  for  this  study  was  in  the  early  stages  of 
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the  P&D  phase.  The  B/CR  valuation  reflects  an  impressive  value  added 
between  the  cost  of  implementing  TRA  and  knowledge-based  practices, 
and  the  potentially  derived  benefits.  About  $2.40  of  benefit  is  gained  or 
saved  for  every  dollar  expended  (i.e.,  efficiency).  The  ROI  percent  valua¬ 
tion  indicates  an  approximate  return  of  144.9  percent  on  the  program’s 
investment  in  TRA  practices,  or  approximately  $1.44  saved  for  every 
dollar  invested.  The  NPV  valuation  result  is  approximately  $782.2  mil¬ 
lion.  The  BEP  valuation  reflects  being  able  to  recoup  (in  efficiency  gains) 
the  full  initial  cost  of  the  investment  in  TRA  practices  in  approximately 
4.5  years.  Finally,  the  ROA  valuation  results  in  an  estimated  return 
of  $1,166.3  million,  which  is  $384.1  million  more  value  than  the  NPV 
estimate,  and  an  estimated  return  of  $468.5  million  above  the  cost  of 
implementing  TRA  practices.  In  this  example  of  the  MUOS  program, 
all  of  the  key  ROI  metrics  suggest  a  sufficiently  mature  technology  base. 
Furthermore,  it  indicates  the  technological  risk  of  proceeding  into  the 
next  development  phase  is  low. 

Analysis  of  JASSM-ER.  The  Air  Force’s  Joint  Air-to-Surface 
Standoff  Missile-Extended  Range  (JASSM-ER)  program  will  “field  a 
next-generation  cruise  missile  capable  of  destroying  the  enemy’s  war- 
sustaining  capability  from  outside  its  air  defenses.  The  JASSM-ER 
missiles  are  low-observable,  subsonic,  and  have  a  range  greater  than  500 
miles”  (GAO,  2012,  p.  91).  The  program  acquisition  state  for  this  study 
was  in  the  early  stages  of  the  P&D  phase.  Similar  to  the  MUOS  program, 
the  resultant  economic  valuations  are  equally  impressive.  Of  particular 
note  here,  the  ROA  valuation  results  in  an  estimated  return  of  $806.5 
million,  which  is  $229.6  million  more  value  than  the  NPV  estimate,  and 
an  estimated  return  of  $432.9  million  above  the  cost  of  implementing 
TRA  practices.  Similar  to  the  MUOS  program,  all  of  the  JASSM-ER 
program  key  ROI  metrics  suggest  a  sufficiently  mature  technology  base. 
It  also  indicates  the  technological  risk  of  proceeding  into  the  next  devel¬ 
opment  phase  is  low. 

Summary  of  Data  Analysis 

Using  our  model,  the  data  from  the  GAO  (2012)  report  on  the  2012 
MDAP  portfolio  were  first  sorted  by  risk  percent  in  ascending  order. 
The  data  were  then  filtered  by  military  Service.  Figures  1,  2,  and  3  pro¬ 
vide  illustrative  histograms  of  each  of  the  Services’  programs  that  have 
been  highlighted  in  this  article.  The  first  major  finding  revealed  by  this 
analysis,  and  consistent  across  the  portfolios  of  the  military  Services, 
was  that  ROI  percent  decreases  as  program  risk  and  cost  increase.  This 
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coincides  with  results  from  other  studies:  larger  programs  are  inherently 
more  complex  and  risk-prone  than  smaller,  shorter  duration  programs, 
which  have  exhibited  as  much  as  a  90  percent  success  rate  (Benediktsson 
&  Dalcher,  2005).  In  addition,  increasing  risk  percent  indicates  decreas¬ 
ing  technology  maturity;  consequently,  programs  with  a  larger  number 
of  unstable  and  immature  technologies  will  have  a  larger  risk  and  lower 
ROI.  The  most  significant  finding  is  that  ROA  increases  as  risk  increases 
and  ROI  percent  decreases,  especially  if  risk-reducing  acquisition  prac¬ 
tices  are  used,  such  as  evolutionary  acquisition,  dividing  acquisitions 
into  smaller  increments,  and  spiral  development  (Benediktsson  & 
Dalcher,  2005;  Reagan  &  Rico,  2010).  Delaying  a  program  due  to  size 
and  technology  instability  and  immaturity  by  dividing  the  scope  into 
numerous  smaller  increments,  spirals,  and  iterations  across  the  entire 
acquisition  life  cycle  may  result  in  greater  economic  benefits  for  each 
military  Service.  This  supports  the  concept  provided  by  other  studies, 
which  reflect  that  when  there  is  heightened  risk,  the  flexibility  to  delay 
a  decision  or  investment  can  be  quite  valuable  (Dixit  &  Pindyck,  1995; 
Fichman,  Keil,  &  Tiwana,  2005;  Luehrman,  1995;  Trigeorgis,  1993). 


FIGURE  1.  ILLUSTRATIVE  ARMY  RISK,  ROI,  AND  ROA  DATA  FROM 
2012  MDAP  PORTFOLIO 


iv  rpi ii  i 

i  p  i  i  iiiiii 
riii  i~ihii 


Source:  GAO,  2012 

It  canbe  seen  from  the  earlier  examples  thatthe  use  of  classical  economic 
valuation  methods  may  provide  useful  management  insight  into  the 
state  of  an  acquisition  program.  In  addition,  ROA  may  provide  each 


Defense  ARJ,  October  2014,  Vol.  21  No.  4: 826-850 


840 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


military  Service  a  useful  estimate  of  the  value  of  deferring  a  program 
until  its  technologies  are  sufficiently  mature,  even  when  NPV  indicates 
no  further  investment  may  be  warranted,  hence  our  motivation  for 
including  ROA  in  our  process  framework  (Kodukula  &  Papudesu,  2006). 

FIGURE  2.  ILLUSTRATIVE  NAVY  RISK,  ROI,  AND  ROA  DATA  FROM 
2012  MDAP  PORTFOLIO 


1.2 


IDECM  NMT  VTUAV  SSC  MUOS  BAMS  E-2DAHE 


Source:  GAO,  2012 


Delaying  a  program  due  to  size  and  technology 
instability  and  immaturity  by  dividing  the  scope 
into  numerous  smaller  increments,  spirals,  and 
iterations  across  the  entire  acquisition  life  cycle 
may  result  in  greater  economic  benefits  for  each  of 
the  military  services. 
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FIGURE  3.  ILLUSTRATIVE  AIR  FORCE  RISK,  ROI,  AND  ROA  DATA 
FROM  2012  MDAP  PORTFOLIO 


Source:  GAO,  2012 


Trend  Analysis  (2003-2012) 

As  we  extended  our  study  of  the  GAO  data  to  encompass  the  2003- 
2012  period,  our  analysis  indicated  what  appears  to  be  inconsistency 
in  the  level  of  adherence  and  commitment  by  the  individual  military 
Services  in  their  execution  of  knowledge-based  acquisition  practices 
mandated  by  Congress  and  the  DoD  (U.  S.  General  Accounting  Office, 
2003,  2004;  GAO,  2005,  2006,  2007,  2008a,  2009,  2010,  2011,  2012). 
Specifically,  as  reflected  in  Table  8,  we  found,  on  average,  taht  40  percent 
of  the  Army's  MDAP  CTs  rated  as  sufficiently  mature,  compared  to  57 
percent  for  the  Navy  and  67  percent  for  the  Air  Force.  Perhaps  even  more 
telling  is  that  the  commitment  level  of  adherence  to  TRA  knowledge- 
based  practices  appears  to  have  carried  through  to  the  level  of  acquisition 
performance  success  realized  during  this  decade.  This  seems  to  be  con¬ 
sistent  with  results  from  other  studies  by  the  GAO  and  DoD,  showing 
that  technology  maturity,  or  lack  thereof,  is  a  predictor  of  acquisition 
performance  (DoD,  2008;  GAO,  2008b).  Moreover,  weapon  systems  that 
use  mature  technologies  will  have  better  acquisition  performance  than 
those  using  immature  technologies. 
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TABLE  8.  2003-2012  DoD  MILITARY  SERVICES  CT  MATURITY 
ASSESSMENTS 


Year 

Army  Critical 
Technologies 

Navy  Critical 
Tehnologies 

Air  Force  Critical 
Technologies 

Immature 

Total 

Mature 

Immature 

Total 

Mature 

Immature 

lotal 

Mature 

2012 

35 

83 

57.8% 

34 

109 

68.8% 

25 

74 

66.2% 

2011 

44 

122 

63.9% 

31 

102 

69.6% 

29 

82 

64.6% 

2010 

43 

106 

59.4% 

35 

104 

66.3% 

8 

72 

88.9% 

2009 

83 

124 

33.1% 

40 

100 

60% 

18 

82 

78% 

2008 

81 

128 

36.7% 

48 

117 

59% 

30 

96 

68.8% 

2007 

91 

125 

27.2% 

71 

160 

55.6% 

44 

108 

59.3% 

2006 

99 

140 

29.3% 

63 

103 

38.8% 

44 

98 

55.1% 

2005 

104 

118 

11.9% 

76 

130 

41.5% 

47 

105 

55.2% 

2004 

85 

109 

22% 

60 

102 

41.2% 

30 

102 

70.6% 

2003 

7 

19 

63.2% 

10 

33 

69.7% 

11 

37 

70.3% 

Avg 

67 

107 

40.451% 

47 

106 

57.061% 

29 

86 

67.7% 

Source:  U.S.  General  Accounting  Office,  2003,  2004;  GAO,  2005,  2006,  2007,  2008a, 
2009,  2010,  2011,  2012 


For  each  of  the  military  Services,  and  the  DoD  overall— including  the 
Marines  and  Missile  Defense  Agency— a  few  trends  seemingly  emerge 
as  reflected  in  Table  9:  (a)  an  overall  lowering  of  risk  percent  in  the 
weapon  system  portfolio,  which  indicates  an  overall  reduction  in  the 
incorporation  of  immature  technology  into  development  programs; 
(b)  an  overall  improvement  in  B/CR,  indicating  growth  in  execution 
efficiency;  (c)  overall  growth  in  ROI  percent,  indicating  a  trend  of 
maximizing  return  on  technology  choices;  (d)  improvement  in  BEP, 
indicating  less  time  needed  before  the  benefits  of  technology  readiness 
exceed  costs;  and  (e)  an  overall  lowering  of  ROA  valuation  as  risk 
percent  is  lowered,  indicating  more  programs  are  waiting  for  critical 
technologies  to  mature  before  entering  into  development.  The  trends 
seem  to  suggest  that  the  incorporation  of  TRAs  and  knowledge-based 
practices  into  the  acquisition  programs  of  each  military  Service 
may  indeed  improve  cost,  schedule,  and  technical  performance 
of  those  programs,  and  consequently  of  the  overall  DoD  weapon 
systems  portfolio. 
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TABLE  9.  SUMMARY  ROI  OF  TRA  ANALYSIS  OF  U.S.  GAO  MDAP 
DATA  FROM  2003-2012 


Yr. 

No. 

Critical 

Risk 

Cost 

Benefit 

B/ 

ROI 

NPV 

BEP 

ROA 

Pgms 

Technologies 

% 

CR 

(Yts) 

Immature 

Tot 

Mature 

2012 

47 

103 

345 

70.1% 

8.0% 

$82,935.8 

$228,547.6 

2.8:1 

175.6% 

$114,962.5 

3.6 

$163,957.1 

2011 

49 

106 

371 

71.4% 

8.7% 

$83,456.8 

$228,483.7 

2.7:1 

173.8% 

$114,386.1 

3.6 

$163,487.4 

2010 

55 

105 

372 

71.8% 

9.6% 

$87,909.6 

$238,416.2 

2.7:1 

171.2% 

$118,533.9 

3.7 

$169,952.1 

2009 

59 

177 

420 

57.9% 

10.2% 

$97,444.3 

$262,715.5 

2.7:1 

169.6% 

$130,039.9 

3.7 

$186,825.9 

2008 

72 

208 

466 

55.4% 

8.7% 

$106,304.8 

$291,048.4 

2.7:1 

173.8% 

$145,712.6 

3.6 

$208,258.1 

2007 

62 

241 

451 

46.6% 

10.0% 

$87,997.8 

$237,794.1 

2.7:1 

170.2% 

$117,907.0 

3.7 

$169,261.4 

2006 

51 

225 

428 

47.4% 

11.2% 

$84,425.8 

$225,533.7 

2.7:1 

167.1% 

$110,862.8 

3.8 

$159,782.8 

2005 

54 

251 

443 

43.3% 

11.3% 

$80,422.3 

$214,634.1 

2.7:1 

166.9% 

$105,428.4 

3.8 

$152,001.2 

2004 

51 

193 

391 

50.6% 

10.9% 

$67,429.3 

$180,664.2 

2.7:1 

167.9% 

$89,007.0 

3.8 

$128,150.2 

2003 

26 

39 

117 

66.7% 

12.0% 

$47,702.0 

$126,332.6 

2.6:1 

164.8% 

$61,688.8 

3.9 

$89,182.2 

Note.  All  costs  and  benefits  are  in  millions  of  dollars. 

Source:  U.S.  General  Accounting  Office,  2003,  2004;  GAO,  2005,  2006,  2007,  2008a, 
2009,  2010,  2011,  2012 


Conclusions 

For  several  decades  the  DoD  and  Congress  have  endeavored  to  insti¬ 
tute,  revamp,  refine,  tweak,  overhaul,  and  reform  the  Defense  Acquisition 
System  in  attempts  to  structure  a  system  of  procuring  major  weapon 
systems  as  efficiently,  effectively,  and  affordably  as  possible,  but  unfor¬ 
tunately,  without  achieving  significant  sustained  improvement  (Bair, 
1994;  DoD,  2013b;  Fox,  2011;  U.  S.  General  Accounting  Office,  1988). 
Acquisition  performance  continues  to  struggle  and  manifest  itself  in 
the  form  of  cost  and  schedule  overages,  and  reduced  DoQs.  Economic 
evidence,  however,  is  starting  to  emerge  indicating  that  investments  in 
knowledge -based  practices,  especially  TRAs  as  a  means  of  achieving 
technology  maturity,  are  beginning  to  pay  off.  In  this  article,  we  have 
introduced  a  model  to  evaluate  the  costs  and  benefits  of  the  current 
MDAP  portfolios  of  the  U.S.  Army,  Navy,  and  Air  Force  using  classic 
economic  techniques  such  as  ROI,  NPV,  and  ROA.  We  have  shown  there 
is  added  ROI  valuation  due  to  the  use  of  TRAs  for  MDAPs.  We  have  also 
shown  that  the  ability  to  delay  a  decision  to  move  into  development/ 
production  until  CTs  (and  associated  risk)  are  sufficiently  matured 
(mitigated)  may  provide  significant  cost  benefit  to  a  program.  We  have 
defined  a  set  of  valuation  metrics  for  ROI  of  TRAs  that  includes  costs. 
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benefits,  B/CR,  ROI  percent,  NPV,  BEP,  and  ROA.  Indeed,  used  along 
with  traditional  discounted  cash-flow  methods,  real  options  analysis 
provides  additional  insight  for  the  decision  maker  into  the  cost  and  tech¬ 
nology  risk  for  MDAPs.  In  addition,  use  of  the  TRA  framework  enhances 
opportunities  to  maximize  ROI  from  new,  complex  technologies  targeted 
for  MDAPs.  The  ROA  measure  supports  (a)  valuation  of  the  decision  to 
delay,  (b)  identification  and  quantification  of  risk  associated  with  CTs, 
and  (c)  prioritization  of  program  development  and  mitigation  of  risks. 


Acquisition  performance  continues  to  struggle 
and  manifest  itself  in  the  form  of  cost  and  schedule 
overages  and  reduced  DoQs.  Economic  evidence, 
however,  is  starting  to  emerge  indicating  that 
investments  in  knowledge-based  practices, 
especially  TRAs  as  a  means  of  achieving  technology 
maturity,  are  beginning  to  pay  off. 


Our  study  has  also  revealed  an  inconsistency  between  the  military 
Services  in  their  commitment  level  of  adherence  to  knowledge-based 
practices  as  mandated  by  DoDI  5000.02  (DoD,  2013e).  In  particular, 
although  the  evidence  continues  to  mount  indicating  that  programs 
with  immature  technology  experience  cost,  schedule,  and  performance 
shortfalls,  the  military  Services  appear  to  discount  this  risk,  continuing 
to  allow  immature  technology  into  their  development  programs.  Table 
8  speaks  clearly  to  this  issue,  showing  that,  on  average  over  the  past 
decade,  40  percent  of  the  Army's  MDAP  CTs  rated  as  sufficiently  mature, 
compared  to  57  percent  for  the  Navy  and  67  percent  for  the  Air  Force. 

This  article  is  designed  to  help  decision  makers  within  the  Army, 
Navy,  and  Air  Force,  as  well  as  the  DoD  overall,  in  understanding  the 
economic  impact  of  the  use  or  nonuse  of  TRA  knowledge-based  practices. 
Through  objective,  quantifiable  measures  of  performance  such  as  those 
discussed  in  this  article,  we  can  begin  making  significant  strides  toward 
improving  the  outcome  of  our  investments  in  major  weapon  systems 
acquisition  (Weapon  Systems  Acquisition  Reform  Act,  2009). 
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The  Human  Viewpoint 
Approach  for  Architecting 
and  Acquisition 

f  Holly  A.  H.  Handley  and  Beverly  G.  Knapp 


The  U.S.  Department  of  Defense  Architecture  Framework 
(DoDAF)  provides  a  standard  framework  for  transforming 
systems  concepts  into  a  consistent  set  of  products 
containing  the  elements  and  relationships  required 
to  represent  a  complex  operational  system.  Flowever, 
without  a  human  perspective,  the  current  DoDAF  does  not 
account  for  the  human  performance  aspects  needed  to 
calculate  the  human  contribution  to  system  effectiveness 
and  cost.  The  Fluman  Viewpoint  gives  systems  engineers 
additional  tools  to  integrate  human  considerations  into 
systems  development  by  facilitating  identification  and 
collection  of  human-focused  data.  It  provides  a  way  to 
include  Fluman  Systems  Integration  (HSI)  constructs 
into  mainstream  acquisition  and  systems  engineering 
processes  by  promoting  early,  frequent  coordination  of 
analysis  efforts  by  both  the  systems  engineering  and 
HSI  communities. 
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The  U.S.  Department  of  Defense  Architecture  Framework  (DoDAF) 
is  used  by  the  engineering  and  acquisition  communities  to  describe  the 
overall  structure  for  designing,  developing,  and  implementing  systems 
(DoDAF  Working  Group,  2004).  DoDAF  provides  a  standard  framework 
for  transforming  systems  concepts  into  a  consistent  set  of  products 
that  contains  the  elements  and  relationships  required  to  represent  a 
complex  operational  system.  The  use  of  an  architecture  framework, 
such  as  DoDAF,  in  the  acquisition  process  can  be  a  critical  enabler  for 
systems  success  since  it  provides  a  structured  approach  to  identifying 
and  addressing  technical  issues  early  in  the  systems  life-cycle  process. 

Background 

DoDAF  was  designed  to  meet  the  needs  of  multiple  stakeholders, 
including  program  managers,  systems  engineers,  and  acquisition  execu¬ 
tives.  The  architecture  framework  can  be  used  to  provide  pertinent 
information  to  different  communities  by  employing  various  viewpoints. 
Each  viewpoint  is  built  by  extracting  data  focused  on  a  specific  facet 
of  the  system  and  displaying  it  to  the  user  through  a  set  of  models. 
Models  can  be  documents,  spreadsheets,  dashboards,  or  other  graphical 
representations  that  organize  and  display  system  data.  This  allows 
users  to  focus  on  specific  areas  of  interest,  such  as  capabilities,  data  and 
information,  projects,  services,  and  standards,  among  other  viewpoints 
(DoDAF  Working  Group,  2010).  However,  noticeably  missing  from  the 
list  of  viewpoints  is  one  that  focuses  on  the  human  perspective:  the 
Human  Viewpoint. 

DoDAF  is  fundamentally  about  creating  a  set  of  models  representing 
the  system  to  enable  effective  decision  making  to  support  systems  engi¬ 
neering  and  acquisition  processes.  However,  without  including  models 
that  focus  on  the  human  perspective,  the  current  DoDAF  framework  does 
not  account  for  the  human-performance  aspects  needed  to  contribute  to 
systems  effectiveness  and  cost.  Without  this  type  of  information,  there 
is  no  basis  to  make  informed  decisions  about  the  tradeoffs  between  sys¬ 
tems  design  and  human-related  issues  (Knapp  &  Smillie,  2010).  DoDAF 
ensures  that  the  architecture  descriptions  facilitate  the  creation  of 
systems  requirements  that  will  achieve  the  desired  outcomes;  however, 
systems  engineers  currently  do  not  have  sufficient  tools  to  quantitatively 
integrate  human  considerations  into  systems  development  (Hardman  & 
Colombi,  2012). 
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This  article  reviews  the  Human  Viewpoint  and  then  presents  a 
current  methodology  for  identifying  and  capturing  data  in  the  Human 
Viewpoint  models.  The  relationship  between  the  Human  Viewpoint 
and  Human  Systems  Integration  (HSI)  is  then  identified,  and  support 
for  using  the  Human  Viewpoint  in  the  acquisition  process  is  provided. 
Finally,  an  example  of  how  the  Human  Viewpoint  can  be  used  to  capture 
appropriate  human  system  data  to  support  systems  design  decisions  is 
described. 


The  Need  for  a  Human  Perspective 

DoDAF  defines  different  perspectives  or  views  that  logically  combine 
to  describe  a  system  architecture.  A  viewpoint  provides  a  self-contained 
set  of  models  that  provides  a  complete  set  of  data  for  evaluation  con¬ 
sistent  with  the  perspective  of  the  view.  When  DoDAF  was  initially 
released,  HSI  practitioners  argued  that  without  a  viewpoint  that  included 
the  human  component  of  the  system,  there  was  no  basis  in  the  architec¬ 
ture  for  analysis  of  human  issues  that  may  impact  multiple  aspects  of  the 
system  (Hildebrand  &  Adams,  2002).  For  example,  analyses  that  measure 
the  human  impact  on  system  performance;  cost-benefit  analyses  that 
consider  the  influence  of  manpower,  personnel,  and  training  on  total 
costs;  and  requirement  analyses  that  include  the  human  specifications  to 
adequately  operate  and  maintain  the  system  all  require  human-focused 
data— none  of  these  analyses  could  be  performed  with  the  data  currently 
captured  in  the  framework.  With  a  viewpoint  that  captures  human  con¬ 
siderations,  these  factors  could  be  assessed  and  addressed  early  in  the 
acquisition  process,  similar  to  technical  evaluations.  The  consideration 
of  human  issues  can  enhance  overall  systems  performance  by  ensuring 
efficient  and  effective  use  of  human  resources  within  the  system,  ulti¬ 
mately  reducing  the  overall  cost  of  a  system  (Knapp  &  Smillie,  2010). 

Developers  of  the  original  DoDAF  deskbook  made  an  initial  attempt 
to  represent  humans  in  the  Operational  Viewpoint  products  by  includ¬ 
ing  the  role  of  the  human  and  human  activities  associated  with  a  system 
(Hildebrand  &  Adams,  2002).  Likewise,  in  the  recent  version  of  DoDAF 
(version  2.02),  human  components  can  be  identified  under  the  Performer 
construct  in  the  Services  Viewpoint  (DoDAF  Working  Group,  2010). 
While  both  of  these  attempts  allow  the  identification  of  the  human  as  an 
element  of  the  system,  simply  identifying  what  functions  are  allocated  to 
humans  does  not  provide  the  robustness  required  to  evaluate  the  human 
component  and  its  impact  on  the  system;  it  does  not  capture  the  multiple 
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human  attributes  required  to  evaluate  the  ability  of  a  system  to  support 
operational  requirements  and  accomplish  a  mission  with  the  current 
human  configuration.  This  requires  an  integrated  viewpoint,  with  a  set 
of  models  appropriate  for  analysis  from  the  human  perspective. 


The  consideration  of  human  issues  can  enhance 
overall  systems  performance  by  ensuring  efficient 
and  effective  use  of  human  resources  within  the 
system,  ultimately  reducing  the  overall  cost  of  a 
system  (Knapp  &  Smillie,  2010). 


With  a  defined  Human  Viewpoint,  the  role  of  the  human  within  the 
system  is  defined  and  task  activities  are  described  at  a  level  useful  for 
analysis.  Human  characteristics,  limitations,  and  constraints  that  affect 
performance  are  also  included  in  the  models,  as  well  as  human-centered 
coordination  and  metrics.  The  design  of  a  complete  viewpoint  allows 
the  impact  of  the  human  presence  to  be  evaluated  and  may  be  the  driver 
for  change  in  the  other  views.  Without  this  view,  no  basis  exists  in  the 
architecture  for  analysis  and  propagation  of  human  issues  (Handley  & 
Smillie,  2008). 

The  Human  Viewpoint  was  developed  by  an  international  panel  of 
systems  engineering  and  HSI  practitioners  (Handley  &  Smillie,  2008). 
The  goal  was  to  develop  an  integrated  set  of  models,  similar  to  the 
other  viewpoints,  that  organized  human  data  for  use  in  the  architec¬ 
ture  description.  These  models  were  also  linked  to  other  architecture 
components,  through  relationships  with  the  Operational  and  System 
Viewpoints,  to  provide  connections  to  the  overall  system.  The  Human 
Viewpoint  contains  seven  models  that  include  different  aspects  of  the 
human  element,  such  as  roles,  tasks,  constraints,  training,  and  metrics 
(Table  1).  It  also  includes  a  human  dynamics  component  to  capture  tem¬ 
poral  information  pertinent  to  the  behavior  of  the  human  system.  The 
resulting  human  perspective  provides  a  basis  for  stakeholder  decisions 
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regarding  the  human  component  by  linking  the  systems  engineering 
community  to  the  manpower  and  personnel  integration,  training,  and 
human  factors  communities  (Baker,  Pogue,  Pagotto,  &  Greenley,  2006). 


TABLE  1.  HUMAN  VIEWPOINT  MODELS 


Product 

Name 

Description 

HV-A 

Concept 

A  conceptual,  high-level  representation 
of  the  human  component  of  the 
enterprise  architecture  framework. 

HV-B 

Contraints 

Sets  of  characteristics  that  are  used 

to  adjust  the  expected  roles  and 
tasks  based  on  the  capabilities  and 
limitations  of  the  human  in  the  system. 

HV-C 

Tasks 

Descriptions  of  the  human-specific 
activities  in  the  system. 

HV-D 

Roles 

Descriptions  of  the  roles  that 

have  been  defined  for  the  humans 

interacting  with  the  system. 

HV-E 

Human  Network 

The  human-to-human  communication 

patterns  that  occur  as  a  result  of  ad 
hoc  or  deliberate  team  formation, 
especially  teams  distributed  across 
space  and  time. 

HV-F 

Training 

A  detailed  accounting  of  how 
training  requirements,  strategy,  and 
implementation  will  impact  the  human. 

HV-G 

Metrics 

A  repository  for  human-related  values, 
priorities,  and  performance  criteria;  it 
maps  human  factors  metrics  to  any 

other  Human  View  elements. 

HV-H 

Human  Dynamics 

Dynamic  aspects  of  human  systems 
components  defined  in  other  views. 

Note.  Adapted  from  "Architecture  Framework  Human  View:  The  NATO  Approach,"  by 
H.A.H.  Handley  and  R.  J.  Smillie  (2008),  Systems  Engineering,  77(2),  pp.  156-164. 
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Building  a  Human  Viewpoint 


The  original  Human  Viewpoint  was  defined  as  a  set  of  required  prod¬ 
ucts,  but  without  a  prescribed  methodology  to  identify  and  capture  the 
human  data.  More  recent  work  has  identified  how  the  Human  Viewpoint 
models  can  be  compiled  by  following  a  series  of  steps,  broken  into  stages 
(Handley  &  Kandemir,  2013).  Each  stage  represents  the  development  of 
a  critical  human  performance  dimension.  The  first  stage  is  initiated  by 
visually  representing  the  system  concept  of  operations,  using  one  or  more 
diagramming  methods  (e.g.,  concept  map,  systemigram,  rich  pictures, 
etc.).  Use  cases  (HV-A)  are  then  developed  that  describe  the  interaction 
of  humans  with  the  operational  environment  and  system  components. 
The  second  stage  develops  the  human  roles  (HV-D)  and  tasks  (HV-C), 
often  in  tandem.  Tasks  describe  the  human  activities,  usually  by  more 
fully  decomposing  higher  level  functions.  Roles  represent  job  functions 
or  task  groupings.  The  mapping  between  the  two  is  a  key  product  of  the 
development  as  it  drives  manning  and  training  requirements.  These  first 
two  stages  are  shown  in  Figure  1. 


FIGURE  1.  HUMAN  VIEWPOINT  AND  DEVELOPMENT-STAGES  I 
AND  II 
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The  third  stage  focuses  on  human  interactions  and  develops  a  human 
network,  usually  represented  as  a  work  process  (H V-E),  which  describes 
the  interactions  of  the  roles  completing  tasks  to  support  the  use  case. 
This  is  another  key  product  of  the  Human  Viewpoint  as  it  describes 
human  activity  over  time,  which  is  a  driver  of  workload  (and  overload) 
for  the  individual  roles.  At  this  stage,  role  locations  can  also  be  included, 
which  is  important  for  designing  distributed  teams.  Metrics  (HV-G) 
representing  human  performance  criteria  are  also  determined.  Subject 
matter  experts,  often  HSI  practitioners,  are  usually  consulted  at  this 
stage  to  ensure  that  the  human  interactions  with  the  system  are  accu¬ 
rately  represented.  This  stage  is  shown  in  Figure  2. 

FIGURE  2.  HUMAN  VIEWPOINT  DEVELOPMENT-STAGE  III 


Operational 

Documents 


In  the  fourth  stage,  manning  or  crew  assignments  (H  V-BI)  are  com¬ 
pleted  by  mapping  personnel  to  roles  based  on  current  qualifications. 
Additional  training  (HV-F)  requirements  are  determined  based  on 
anticipated  knowledge,  skills,  and  abilities  requirements.  Other  human 
factor  constraints  (HV-BII)  are  captured  that  may  impact  the  human 
system,  such  as  work  cycle  and  availability.  Figure  3  shows  the  completed 
Human  Viewpoint  development  process. 
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FIGURE  3.  HUMAN  VIEWPOINT  DEVELOPMENT-COMPLETED 


After  the  completion  of  the  individual  products,  the  human  dynamics 
(HV-H)  can  be  used  to  pull  together  the  information  captured  in  all  the 
products  to  evaluate  the  total  human  system  behavior. 
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For  example,  an  event  from  the  environment  may  trigger  a  task  (H  V- 
C).  The  role  (HV-D)  responsible  for  the  task  begins  processing  it.  The  role 
may  coordinate  with  team  members  (HV-E)  for  information  exchange 
during  processing.  The  way  the  task  is  processed  may  depend  on  traits 
of  the  actual  person  fulfilling  the  role  (HV-B)  and  training  completed 
(HV-F).  Use  of  a  system  resource  (HV-C)  to  complete  the  task  can  also  be 
included.  Additionally,  other  constraints  such  as  human  characteristics 
and  health  hazards  (HV-B)  may  moderate  the  performance  of  the  task. 
Once  the  task  is  completed,  metrics  (HV-G)  are  used  to  evaluate  the  task 
performance  (Handley  &  Smillie,  2010). 

The  Human  Viewpoint  models  should  capture  information  about  all 
personnel  who  interact  with  the  system  in  any  capacity.  The  operators, 
maintainers,  and  support  personnel  possess  particular  knowledge,  skills, 
and  abilities  that  must  be  accounted  for  in  the  system  design  along  with 
their  physical  characteristics  and  constraints,  just  as  the  technology 
elements  of  the  system  have  inherent  capabilities  and  constraints.  The 
inclusion  of  the  human  component  in  the  architecture  is  essential  to 
ensure  efficient  interfaces  between  technology  elements  and  the  system's 
intended  users,  as  well  as  the  fit  to  their  physical  characteristics. 

The  initial  Human  Viewpoint  development  was  done  as  a  "prod¬ 
uct-based"  approach,  that  is  the  viewpoint  was  designed  as  a  set  of 
architecture  products  that  captured  the  elements  representing  the  inter¬ 
action  of  the  human  with  the  system.  These  products  were  aligned  with 
the  other  DoDAF  Version  1.0  viewpoints  and  were  specifically  designed 
to  extend  existing  DoDAF  products  wherever  possible.  For  example,  ele¬ 
ments  such  as  "task"  or  "role"  can  be  derived  from  a  further  refinement 
of  data  already  captured  in  the  DoDAF  Version  1.0  products.  However, 
DoDAF  Version  2.0  (initially  released  in  2009)  is  a  data-based  approach 
with  a  focus  on  capturing  the  data  needed  for  a  system,  and  products  or 
views  are  rendered  as  needed  from  the  data  for  decision  making  or  sys¬ 
tem  design  considerations  (DoDAF  Working  Group,  2010).  The  Human 
Viewpoint  was  aligned  with  the  DoDAF  Version  2.0  Meta  Model  (DM2) 
to  produce  "Fit  for  Purpose"  views.  These  views  can  be  used  to  augment 
the  standard  sets  of  architectural  products  with  human-centered  infor¬ 
mation  important  to  the  system  description.  See  Handley  (2012a)  for  a 
complete  description  of  the  implementation  of  the  Human  Viewpoint 
with  DoDAF  Version  2.0. 
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The  Link  Between  Systems  Engineers  and 
Human  Systems  Integration 

HSI  is  a  disciplined,  unified,  and  interactive  approach  that  integrates 
human  considerations  into  systems  design  to  improve  total  system  per¬ 
formance  and  reduce  costs  of  ownership  (Cochrane  &  Hagan,  2001).  It 
is  also  a  strategy  to  integrate  the  multiple  domains  of  Human  Factors 
Engineering,  Training,  Manpower,  Personnel,  Health  Hazards  and 
System  Safety.  These  domains  collectively  define  how  the  human  compo¬ 
nent  will  impact  systems  performance  (e.g.,  mission  achievement,  safety, 
and  cost),  and  also  define  how  the  system  impacts  the  human  component, 
as  reflected  in  skill  gaps  and  training  requirements,  manning  levels,  and 
workload  (Baker  et  al.,  2006).  HSI  ensures  that  the  needs  of  the  human 
user  are  considered  throughout  the  system  acquisition  process  and  life 
cycle,  but  it  represents  a  departure  point  for  current  architecture  frame¬ 
works,  as  these  human  considerations  are  not  captured  in  the  standard 
DoDAF  viewpoints. 

The  Human  Viewpoint  can  provide  the  data  and  relationships  nec¬ 
essary  to  address  HSI  concerns  that  are  lacking  in  current  architecture 
frameworks.  For  example,  the  Human  Viewpoint  can  evaluate  the  antici¬ 
pated  impact  of  a  new  system  development  on  the  number  and  type  of 
personnel  required;  the  requisite  knowledge,  skills,  and  abilities  of  the 
personnel;  and  the  anticipated  training  that  will  be  necessary  to  achieve 
proficiency.  To  maximize  task  performance,  which  affects  system  per¬ 
formance,  information  on  human  characteristics  as  well  as  impacts  to 
safety  and  health  hazards  should  be  included  in  the  design,  development, 
and  evaluation  of  the  new  system.  The  Human  Viewpoint  assists  in 
influencing  the  architecture  framework  from  a  "people"  perspective— it 
identifies  the  effect  on  the  development  of  the  workforce  and  changes 
to  their  working  environment  by  identifying  the  roles,  and  therefore 
personnel,  that  are  affected  and  the  requirements  that  are  necessary 
to  transition  the  workforce  and  their  workstations  to  the  future  system 
(Hewitt,  2010). 

The  Human  Viewpoint  gives  systems  engineers  an  additional  tool  to 
integrate  human  considerations  into  systems  development  by  facilitat¬ 
ing  the  identification  and  collection  of  human  component  data  that  can 
be  used  to  improve  systems  design.  The  increase  in  the  complexity  of 
systems  and  the  missions  they  support  heighten  the  need  for  HSI  to  be 
considered  early  in  systems  development.  Ultimately,  the  goal  of  HSI  is 
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to  integrate  considerations  of  human  capabilities  and  limitations  into  the 
design  decision-making  process,  similar  to  what  is  done  for  hardware  and 
software— integration  of  HSI  analysis  into  the  acquisition  and  systems 
engineering  process  is  the  key  to  achieving  this  goal  (Pharmer,  2007). 
The  human— the  most  important  and  unique  system  within  the  system 
of  systems— can  also  be  the  weakest  link  or  highest  risk  in  that  system; 
therefore,  expressing  the  capabilities  and  limitations  of  the  human  in 
the  system  is  imperative  (Baker  et  al.,  2006).  By  developing  the  Human 
Viewpoint  to  be  tightly  coupled  with  the  DoDAF,  the  Human  Viewpoint 
provides  hooks  to  include  HSI  in  the  evolving  systems  concept. 


The  Human  Viewpoint  assists  in  influencing 
the  architecture  framework  from  a  "people" 
perspective— it  identifies  the  impact  on  the 
development  of  the  workforce  and  changes  to  their 
working  environment. 


HSI  is  practiced  across  the  Services,  with  slightly  different  defini¬ 
tions  for  the  set  of  domains.  The  Army  has  taken  the  lead  in  furthering 
the  development  of  the  Human  Viewpoint  and  has  completed  the  first 
steps  to  integrate  it  into  procedures  and  apply  it  to  systems  acquisi¬ 
tion.  (MANPRINT,  or  Manpower  and  Personnel  Integration,  is  the 
Army's  term  for  the  implementation  of  HSI.)  HSI  policy  information  is 
shared  among  the  Services  through  the  Joint  HSI  Working  Group  (2012), 
which  provides  a  venue  for  inter-Service  collaboration  to  support  DoD 
HSI  initiatives. 

Applying  the  Human  Viewpoint  in  Acquisition 

The  Human  Viewpoint  captures  human  systems  data  in  a  program¬ 
matic  way  that  closely  aligns  with  systems  engineering  approaches. 
This  not  only  supports  collaboration  between  the  systems  engineering 
and  HSI  communities,  but  helps  support  the  HSI  objectives  of  informing 
tradeoff  analysis;  in  fact,  one  of  the  original  drivers  for  the  development 
of  the  Human  Viewpoint  was  the  concern  that  the  DoDAF  views  were 
insufficient  to  address  HSI  issues.  By  explicit  modeling,  the  human 
elements  can  be  considered  early  and  related  closely  to  the  design  and 
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implementation  of  technology  (Bruseberg,  2009).  In  this  way  the  Human 
Viewpoint  models  are  appropriate  inputs  to  the  acquisition  of  complex 
systems. 

The  application  of  DoDAF  and  the  Human  Viewpoint  architec¬ 
ture  products  is  suited  to  different  phases  of  the  Defense  Acquisition 
System  (DoD,  2013).  The  Human  Viewpoint  models  can  inform  the  Joint 
Capabilities  Integration  and  Development  System  (JCIDS)  analysis 
starting  before  Milestone  A  as  capability  gaps  and  approaches  to  desired 
end  states  are  identified.  Functional  requirements  emerge  by  progress¬ 
ing  through  the  Functional  Area  Analysis  (FAA),  Functional  Needs 
Analysis  (FNA),  and  the  Functional  Solution  Analysis  (FSA;  Chairman 
of  the  Joint  Chiefs  of  Staff,  2012).  Manpower,  personnel,  and  training 
options  can  be  explored  for  the  conceptual  system  by  including  the 
human  data  from  the  Human  Viewpoint.  Table  2  shows  the  individual 
Human  Viewpoint  models  that  support  the  JCIDS  process. 
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TABLE  2.  SUPPORT  OF  HUMAN  VIEWPOINT  PRODUCTS 
FOR  JCIDS 


JCIDS  Step 

Goal 

Supporting  Human 
Viewpoint  Models 

Functional 

Area  Analysis 
(FA  A) 

Tasks  to  be 

accomplished 

•F1V-A  provides  an  overview  of 
objectives 

•HV-C  provides  insights  into 
tasks  that  are  required  to 
achieve  military  objectives 
•HV-G  provides  performance 

standards  and  metrics  for 

systems  tasks 

Functional 

Needs  Analysis 
(FNA) 

List  of  capability 

gaps 

•HV-B1  may  identify  manpower 
gaps  that  cannot  be  supported 
by  current  personnel 

•HV-D  identifies  the  needed 
roles  to  support  tasks 
•HV-E  identifies  information 
exchange  requirements 
between  roles-may  also 
identify  implications  of 
distributed  reach-back  teams 


Functional 

Potential  integrated 

•HV-B1  provides  the  ability  to 

Solution 

DOTmLPF-P 

conduct  strategic  manpower 

Analysis  (FSA) 

(Doctrine, 

tradeoffs  and  comparisons 

Organization, 

between  potential  options 

Training,  materiel, 

•HV-B2  identifies  the  impact 

Leadership 

of  personnel  issues  on  career 

and  Education, 

progressions  (as  well  as  costs) 

Personnel,  Facilities- 

•F1V-F  identifies  the  impact  on 

Policy)  Change 

Recommendations 

approach  to 
capability  gaps 

training  programs  (and  costs) 

Post 

Initial  Capabilities 

Complete  set  of  initial  Human 

Independent 
Analysis  (PIA) 

Document  (ICD) 

View  product  documents 
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Using  the  Human  Viewpoint  to  support  the  pre-Milestone  A  out¬ 
comes  facilitates  the  identification  of  HSI  issues  (Baker,  Steward,  Pogue, 
&  Ramotar,  2008).  For  example,  during  the  FAA,  the  HV-C  highlights 
critical  tasks  that  are  most  likely  to  be  assigned  to  humans;  in  the  FNA, 
the  HV-B  and  HV-D  assist  in  the  identification  of  the  current  and  pro¬ 
jected  personnel  required  to  accomplish  those  tasks,  followed  bythe  FSA, 
where  the  HV-F  can  identify  training  requirements  that  may  mitigate  a 
manpower  gap. 

The  Human  Viewpoint  supports  the  Army  MANPRINT  program's 
goals  of  optimizing  total  systems  performance,  reducing  life-cycle  costs, 
and  minimizing  risk  of  soldier  loss  or  injury  by  ensuring  a  systematic 
consideration  of  the  impact  of  the  materiel  design  on  soldiers  through¬ 
out  the  acquisition  process  (Department  of  the  Army,  2001).  Figure  4 
shows  application  of  the  Army  MANPRINT  program,  both  pre-  and 
post-Milestone  A.  The  Human  Viewpoint  products  directly  support  the 
MANPRINT  processes,  which  are  applied  during  pre-Milestone  A,  and 
can  result  in  risk  reduction  and  fewer  changes  in  the  mature  system.  The 
MANPRINT  issue-processing  cycle  (post-Milestone  A)  supports  person¬ 
nel  planning  for  the  deployed  system  by  analyzing  the  work  allocation, 
personnel  demand,  and  required  training.  It  also  allows  early  assessment 
and  mitigation  alternatives  for  personnel  survivability  (i.e.,  force  protec¬ 
tion,  safety,  and  health  hazards). 
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FIGURE  4.  MANPRINT  ACTIVITIES  PRE-  AND  POST-MILESTONE  A 
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automating  critical  tasks  within  functional 
analysis  and  allocation  decisions 
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related  to  Army  readiness  factors 


Assess  MANPRINT 
Issues  in  Design 
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MANPRINT 
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Note.  CBA  =  Capabilities  Based  Assessment;  CDD  =  Capability  Development  Document; 
CPD  =  Capability  Productino  Document;  ICD  =  Initial  Capabilities  Document;  T&E  =  Test 
and  Evaluation;  MANPRINT  =  Manpower  and  Personnel  Integration;  MS  =  Milestone. 


In  short,  HSI  issues  and  systems  requirements  that  impact  the 
human  role  can  be  identified  pre-Milestone  A  (Materiel  Solution 
Analysis)  using  the  Human  Viewpoint.  Then  during  pre-Milestone  B 
(Technology  Maturation  and  Risk  Reduction),  the  FSA  can  be  revisited  to 
assess  the  MANPRINT  implications  of  a  materiel  solution.  For  example, 
changes  to  the  initial  manpower  and  personnel  assessment,  based  on  a 
specific  materiel  option,  can  be  determined  by  examining  the  updated 
architectural  products.  This  may  then  impact  the  expected  training 
requirement,  and  there  may  also  be  updates  to  health  and  safety  issues. 
During  pre-Milestone  C  (Engineering  and  Manufacturing  Development), 
the  Human  Viewpoint  products  should  be  updated  to  align  with  the 
final  HSI  requirements  and  serve  as  an  authoritative  source  for  formal 
test  and  evaluation  activities,  as  well  for  post-Milestone  C  Production 
and  Deployment. 


The  Human  Viewpoint  provides  a  way  to  include  HSI  in  the  main¬ 
stream  acquisition  and  systems  engineering  process  by  promoting  early 
and  frequent  consideration  of  human  roles.  It  also  provides  coordination 
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of  task  analysis  efforts  by  both  systems  engineering  and  HSI  teams. 
Implementing  a  human  perspective  can  significantly  reduce  systems 
risk  due  to  technical  design  problems  by  communicating  information 
about  the  needs  and  constraints  of  the  human  component  and  ensuring 
optimal  performance  and  safety. 


Implementing  a  human  perspective  can 
significantly  reduce  systems  risk  due  to  technical 
design  problems  by  communicating  information 
about  the  needs  and  constraints  of  the  human 
component  and  ensuring  optimal  performance  and 
safety. 


Supporting  Analysis  and  Design 

It  is  not  necessary  to  complete  the  full  set  of  Human  Viewpoint  mod¬ 
els  to  benefit  from  a  human  architecting  effort.  Each  individual  model 
captures  a  "snapshot"  of  different  aspects  of  the  human  system  and  can 
add  value  to  the  architecture  description.  For  example,  the  HV-C  cap¬ 
tures  the  human-level  activities  of  a  system.  These  tasks  can  be  described 
in  terms  of  a  sequence  diagram  (i.e.,  a  temporal  ordering  of  the  tasks). 
This  can  give  an  indication  of  how  well  a  given  sequence  of  tasks  will 
perform,  and  the  performance  predictions  for  alternative  sequences  of 
tasks  can  be  compared.  Analyses  with  single  products  can  also  provide 
insights  by  comparing  "as-is"  and  "to-be"  architectures  (Handley,  2012b). 
For  example,  an  analysis  of  the  role  assignments  (HV-D)  due  to  task 
changes  may  result  in  recommendations  to  reallocate  tasks  to  other  roles 
based  on  workload,  skill  requirements,  or  locations.  For  network-based 
systems,  an  analysis  of  the  HV-E  may  result  in  different  coordination 
requirements  for  distributed  team  members  to  define  responsibilities  and 
information  sharing.  Figure  5  illustrates  the  interactions  between  roles 
on  a  distributed  team  and  identifies  parameters  that  may  be  impacted. 
Even  using  a  subset  of  the  Human  Viewpoint  models  provides  the  oppor¬ 
tunity  to  capture  and  organize  diverse  human  information  to  assess 
design  decisions  and  recommend  improvements. 
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FIGURE  5.  EXAMPLE  OF  A  HUMAN  NETWORK  (HV-E) 


Multiple  Camera  Operators 
May  be  same/different  sensor 


Multiple  Analysts 
May  be  same/c 


different  sensor 

However,  same  number 
of  analysts  and  operators 


PROCEDURES 

Request  camera  change 

SHARED  AWARENESS 


Note.  Adapted  from  "Human  View  Considerations  of  the  Intelligence  Crew  for  the 
Multi-Intelligence  Platform  Long  Endurance,"  by  H.  A.  H.  Handley  and  C.  Kandemir, 
2013,  Alion  Science  &  Technology  Final  Report.  Hr  =  Hour;  OP  =  Operations  Tempo. 


Having  a  dedicated  Human  Viewpoint  allows  evaluation  and 
adjustment  of  the  human  parameters  associated  with  a  system. 
This  analysis  can  be  completed  initially  with  the  data  captured  in 
the  Human  Viewpoint  and  then  associated  with  other  architecture 
viewpoints  for  a  more  comprehensive  analysis.  For  example,  multi¬ 
intelligence,  multisensor  platforms  are  designed  to  carry  a  variety 
of  sensor  types  to  provide  persistent  surveillance  for  long-duration 
missions  (Kerish  &  Perez,  2010).  The  dramatic  increase  in  available 
sensors  over  a  longer  period  of  time  demands  a  more  agile  and  adapt¬ 
able  crew  capable  of  rapidly  processing  sensor  data  from  multiple 
sources.  Because  the  frequency  and  combinations  of  sensors  can 
vary,  the  crew  will  need  to  be  able  to  adjust  to  different  types  and 
combinations  of  sensors  with  minimum  disruption  to  its  organiza¬ 
tional  processes.  The  Human  Viewpoint  can  be  applied  to  generate 
alternative  crew  designs  for  different  sets  of  constraints,  and  then 
evaluate  the  potential  configurations  to  assess  the  organizational  per¬ 
formance.  As  the  sensor  combination  shifts,  personnel  are  reassigned 
to  new  tasks,  based  on  the  constraints  of  required  knowledge,  skills, 
and  abilities,  while  performing  within  an  acceptable  workload  thresh¬ 
old.  For  each  configuration,  both  the  impact  to  the  system  design  and 
compliance  with  HSI  requirements  can  be  evaluated. 


Defense  ARJ,  October  2014,  Vol.  21  No.  4: 852-874 


868 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


In  this  context,  the  Human  Viewpoint  can  be  used  to  evaluate 
Manpower  issues  (the  impact  of  a  fixed  crew  size  responding  to  varied 
task-loading  over  time);  Personnel  issues  (the  impact  of  fixed  special¬ 
ties  responding  to  varied  sensor  types);  and  Human  Factors  issues  (the 
impact  of  operational  tempo  on  task  assignment).  The  Human  Viewpoint 
analyses  can  evaluate  options  such  as  increased  cross  training  and  vary¬ 
ing  skill  levels  to  improve  the  adaptability  of  the  crew  to  meet  system 
needs.  By  identifying  the  attributes  and  parameters  used  to  define  the 
crew,  a  data  map  can  be  created  that  defines  the  data  to  be  captured 
in  each  product,  as  well  as  the  relationships  between  the  variables  of 
interest  (Figure  6).  These  relationships  can  then  be  further  explored  to 
identify  both  limitations  and  opportunities  for  change. 

The  Human  Viewpoint  analysis  of  the  intelligence  crew  supporting 
long- endurance,  multisensor  platforms  facilitated  the  design  of  alter¬ 
native  operator  and  task  arrangements  by  first  capturing  the  human 
systems  requirements  of  the  baseline  configuration.  Next,  the  operator 
requirements  for  different  crew  configurations  were  determined  by 
evaluating  the  roles,  tasks,  and  work  processes  with  different  sets  of 
constraints.  Finally,  a  simulation  model  was  used  to  evaluate  the  effec¬ 
tiveness  of  the  crew  in  the  mission  environment  (Handley  &  Kandemir, 
2013).  After  evaluating  the  impact  of  the  change,  the  candidate  crew 
configuration  was  either  accepted  as  a  viable  alternative,  or  rejected  and 
other  parameter  variances  explored. 

Conclusions 

Humans  play  a  pivotal  role  in  the  performance  and  operation  of  most 
systems,  because  systems  must  be  supported  by  sufficient  manpower,  and 
personnel  must  be  adequately  trained  to  operate  the  system.  Therefore, 
the  absence  of  a  human  perspective  in  the  architecture  framework  leaves 
a  gap  in  both  the  systems  architecting  and  acquisition  processes.  The 
Human  Viewpoint  organizes  information  and  provides  a  comprehen¬ 
sive  and  understandable  representation  of  human  capabilities  related 
to  expected  performance.  It  provides  a  basis  to  inform  stakeholder 
decisions  by  enabling  structured  linkages  between  the  engineering  com¬ 
munity  and  the  HSI  communities.  Finally,  it  provides  a  fully  integrated 
set  of  products  that  can  be  used  to  inform  and  influence  system  design; 
it  facilitates  human  systems  tradeoff  analyses;  and  it  ensures  the  human 
component  has  visibility  as  a  routine  part  of  the  systems  design  and 
acquisition  processes. 
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FIGURE  6.  HUMAN  VIEWPOINT  DATA  MAP 


Note.  AUTL  =  Army  Universal  Task  List;  C3TRACE  =  Command,  Control,  and 
Communications:  Techniques  for  the  Reliable  Assessment  of  Concept  Execution 
Modeling  Environment;  MOS  =  Military  Occupational  Specialty. 
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Review: 

John  F.  Kennedy  had  won  the  1960  Presidential  election  and  entered 
office  with  a  strong  and  growing  Soviet  menace  held  at  bay  by  his  pre¬ 
decessor’s  threat  of  mutual  assured  nuclear  destruction.  The  Cold  War 
strategy  of  containing  communism  also  meant  fighting  surrogate  brush 
wars  and  conducting  bold— sometimes  rash— covert  operations.  Many  of 
these  were  underway  in  Europe,  Southeast  Asia,  and  in  the  Caribbean. 
Vietnam  was  quickly  becoming  a  focal  point  for  U.S.  military  support 
and  intervention  in  this  ideological  battle  of  wills.  For  the  United  States, 
1960-1968  was  a  time  of  strategic  change  abroad  and  brewing  social 
upheaval  at  home.  This  was  the  environment  President  Kennedy  stepped 
into  when  he  took  the  oath  of  office  in  1961. 

Meanwhile,  within  the  Pentagon,  under  the  newly  appointed 
Secretary  of  Defense  Robert  McNamara,  change  would  likewise  become 
the  order  of  the  day.  Supporting  President  Kennedy’s  shift  from  a  mili¬ 
tary  strategy  of  mutual  destruction  to  one  of  “flexible  response”  meant 
moving  away  from  near  total  reliance  on  nuclear  weapons  to  building 
capable  new  conventional  forces  and  weapon  systems.  This  tumultu¬ 
ous  period  of  change  and  refocus  is  the  backdrop  of  Walter  Poole’s  book, 
Adapting  to  Flexible  Response,  1960-1968.  This  important  book  is  the 
second  volume  in  the  acquisition  series  from  the  Office  of  the  Secretary 
of  Defense,  Historical  Office  (released  in  2013). 

Poole  discusses  the  acquisition  of  new  systems  to  support  the  flex¬ 
ible  response  strategy.  Some  of  these  included  producing  and  fielding 
helicopters  in  large  numbers  and  in  direct  combat  roles  for  the  first  time, 
continuing  to  build  nuclear  submarines  and  surface  ships,  and  creating 
fleets  of  aircraft  including  the  F-lll  fighter-bomber  and  heavy  cargo 
lift  C-5A.  To  produce  these  systems,  defense  acquisition  management 
changed  dramatically  under  McNamara’s  Planning,  Programming, 
and  Budgeting  System  and  Five  Year  Defense  Plan.  The  Office  of  the 
Secretary  of  Defense  and  McNamara’s  “whiz  kids”  applied  systems 
analysis  to  requirements  and  acquisitions,  and  encroached  as  never 
before  on  what  had  previously  been  Service  prerogatives. 

Poole’s  book  masterfully  sets  the  stage  for  this  complex  drama  and 
describes  the  forces  inside  and  outside  the  Pentagon  that  drove  defense 
acquisition  during  this  period.  He  then  dives  deeply  into  individual 
weapon  systems  acquisition,  creating  rich  case  studies  that  give  us 
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glimpses  into  the  policies  and  practices  that  went  well— and  those  that 
did  not.  For  instance,  he  compares  the  successful  C-141  with  the  troubled 
C-5A  programs  to  provide  long-range  airlift  and  describes  the  Army’s 
fascinating  political  struggle  to  choose  between  the  M-14  and  the  AR-15 
to  outfit  its  infantry.  He  discusses  Navy  shipbuilding  and  the  love -hate 
relationship  with  Admiral  Hyman  G.  Rickover  and  nuclear  power,  as  well 
as  the  reliability  issues  of  the  Navy’s  “3-T”  missile  (Talos,  Terrier,  and 
Tarter)  and  the  move  toward  a  “standard  missile”  replacement  program. 

Poole’s  tome  is  highly  recommended  reading  for  today’s  acquisition 
professionals.  Many  of  the  challenges  Poole  highlights  from  programs 
in  the  1960s  will  seem  familiar  to  those  encountered  in  today’s  pro¬ 
grams— stringent  requirements,  tight  schedules,  emerging  technologies, 
a  risk-averse  bureaucracy,  and  an  assertive  Congress  that  purports  to 
“help.”  Set  in  a  tumultuous  period  of  evolving  threats,  international  cri¬ 
ses,  domestic  social  unrest,  and  Pentagon  bureaucratic  struggles,  there 
are  important  lessons  to  be  learned  and  insights  to  be  gained  from  Poole’s 
well-written  and  thoroughly  researched  history. 
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but  segregated  (one  to  a  page)  at  the  end  of  the  text.  When  material  is 
submitted  electronically,  each  figure  or  table  should  be  saved  to  a  sepa¬ 
rate,  exportable  file  (i.e.,  a  readable  EPS  file).  For  additional  information 
on  the  preparation  of  figures  or  tables,  refer  to  the  Scientific  Illustration 
Committee,  1988,  Illustrating  Science:  Standards  for  Publication, 
Bethesda,  Maryland:  Council  of  Biology  Editors,  Inc.  Restructure  brief¬ 
ing  charts  and  slides  to  look  similar  to  those  in  previous  issues  of  the 
Defense  ARJ. 

The  author  (or  corresponding  author  in  cases  of  multiple  authors) 
should  attach  a  signed  cover  letter  to  the  manuscript  that  provides  all  of 
the  authors’  names,  mailing  and  e-mail  addresses,  as  well  as  telephone 
and  fax  numbers.  The  letter  should  verify  that  the  submission  is  an 
original  product  of  the  author(s);  that  all  the  named  authors  materially 
contributed  to  the  research  and  writing  of  the  paper;  that  the  submission 
has  not  been  previously  published  in  another  journal  (monographs  and 
conference  proceedings  serve  as  exceptions  to  this  policy  and  are  eligible 
for  consideration  for  publication  in  the  Defense  ARJ);  and  that  it  is  not 
under  consideration  by  another  journal  for  publication.  Details  about 
the  manuscript  should  also  be  included  in  the  cover  letter:  for  example, 
title,  word  length,  a  description  of  the  computer  application  programs, 
and  file  names  used  on  enclosed  DVD/CDs,  e-mail  attachments,  or  other 
electronic  media. 
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righted  manuscripts  that  require  special  posting  requirements  or 
restrictions.  If  we  do  publish  your  copyrighted  article,  we  will  print  only 
the  usual  caveats.  The  work  of  federal  employees  undertaken  as  part  of 
their  official  duties  is  not  subject  to  copyright  except  in  rare  cases. 
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Web-only  publications  will  be  held  to  the  same  high  standards  and 
scrutiny  as  articles  that  appear  in  the  printed  version  of  the  journal  and 
will  be  posted  to  the  DAU  Web  site  at  www.dau.mil. 

In  citing  the  work  of  others,  please  be  precise  when  following  the 
author-date-page  number  format.  It  is  the  contributor’s  responsibility  to 
obtain  permission  from  a  copyright  holder  if  the  proposed  use  exceeds 
the  fair  use  provisions  of  the  law  (see  U.S.  Government  Printing  Office, 
1994,  Circular  92:  Copyright  Law  of  the  United  States  of  America,  p.  15, 
Washington,  D.C.).  Contributors  will  be  required  to  submit  a  copy  of  the 
writer’s  permission  to  the  managing  editor  before  publication. 

We  reserve  the  right  to  decline  any  article  that  fails  to  meet  the 
following  copyright  requirements: 

•  The  author  cannot  obtain  permission  to  use  previously 
copyrighted  material  (e.g.,  graphs  or  illustrations)  in  the 
article. 

•  The  author  will  not  allow  DAU  to  post  the  article  in  our 
Defense  ARJ  issue  on  our  Internet  homepage. 

•  The  author  requires  that  usual  copyright  notices  be  posted 
with  the  article. 

•  To  publish  the  article  requires  copyright  payment  by  the 
DAU  Press. 
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•  Save  image  at  300  dpi  (dots  per  inch)  high-quality  JPEG  or 
Tiff  file 

•  Save  image  at  no  less  thn  5x7  in  front  of  a  plain  background 

•  Men  wear  business  dress  (shirt,  tie,  and  jacket) 

•  Women  wear  appropriate  business  attire 

•  Active  duty  military  wear  Class  A  uniform 

Please  note:  Images  from  the  Web,  Microsoft  PowerPoint,  or  Word 
will  not  be  accepted  due  to  low  image  quality. 

•  One  copy  of  the  typed  manuscript,  including: 

°  Title  (12  words  or  less) 

°  Abstract  of  article  (150  words  or  less) 

°  Two -line  summary 

°  Keywords  (5  words  or  less.  Do  not  include  words  in  the 
title.) 

°  Document  excluding  abstract  and  references  (4,500 
words  or  less  for  the  printed  edition  and  10,000  words 
or  less  for  the  online-only  content) 

These  items  should  be  sent  electronically,  as  appropriately  labeled 
files,  to  Defense  ARJ  Managing  Editor  Norene  Fagan-Blanch  at: 
Norene.Fagan-Blanch@dau.mil. 
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Author  Due  Date 

Publication  Date 
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January  2nd 
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In  most  cases,  the  author  will  be  notified  that  the  submission  has 
been  received  within  48  hours  of  its  arrival.  Following  an  initial  review, 
submissions  will  be  referred  to  referees  and  for  subsequent  consideration 
by  the  Executive  Editor,  Defense  ARJ. 
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